Systems and methods for a chronological-based search engine

ABSTRACT

Methods, apparatus, and database management systems are disclosed for providing, organizing, and present database search results and, more specifically, systems and methods for a chronological-based search engine. One method includes presenting live image data being captured by a camera of the end-user device. The method also includes determining a pose of the end-user device. The pose including a location of the end-user device. Additionally, the method includes sending the pose of the end-user device to an augmented reality platform entity and receiving closed captioning data from the augmented reality platform entity. The closed captioning data is based on the location of the end-user device and includes indications of first topics within the closed captioning data. The method includes superimposing the closed captioning data onto the live image data.

RELATED APPLICATIONS

This application is a U.S. National Stage Entry of InternationalApplication No. PCT/US2017/064749 filed Dec. 5, 2017, which claims thebenefit of U.S. Provisional Application No. 62/430,584, filed Dec. 6,2016 and U.S. Provisional Application No. 62/556,916, filed Sep. 11,2017, each of which are incorporated by reference in their entirety.

TECHNICAL FIELD

The present disclosure generally relates to configuring computingdevices to provide, organize, and present search engine results and,more specifically, systems and methods for a chronological-based searchengine.

BACKGROUND

Search engines index content (e.g., text, images, videos, etc.) storedin databases and/or at network addresses (such as websites on the WorldWide Web.). The indexing process parses and stores metadata associatedwith the content along with a location (sometimes referred to as aUniform Resource Identifier (URI)) at which to retrieve the content.Search engines provide an interface to facilitate users searching forcontent of interest by entering words or phrases into a search box. Thesearch engine searches the metadata generated for the index content todetermine which one(s) of the index content is/are relevant to theentered words or phrases. The search engine provides the user with theURI of the relevant content. Often, the URI are presented to the usedbased on a relevancy score generated while the search engine searchesthe metadata.

SUMMARY

The appended claims define this application. The present disclosuresummarizes aspects of the embodiments and should not be used to limitthe claims. Other implementations are contemplated in accordance withthe techniques described herein, as will be apparent to one havingordinary skill in the art upon examination of the following drawings anddetailed description, and these implementations are intended to bewithin the scope of this application.

An example method includes generating records for entries from aplurality of database content providers, the records identifying topicsincluded in the entries and dates associated with the entries. Theexample method also includes analyzing the entries from the plurality ofdatabase content providers to determine affinities between the topics.In response to receiving a query that includes a search topic from anend-user device, the example method includes (a) retrieving, from adatabase, records identifying the search topic and related topics thathave affinity with the search topic, (b) organizing the records intochronological categories based on the dates associated with thecorresponding entries, each one of the chronological categoriesrepresenting a different period of time, (c) for each of thechronological categories, generating a query result with the processor,the query result organizing the records into hierarchies based onattributes of the entries associated with the records, and (d) sendingthe query result to the end-user device.

An example method includes presenting, on a display of an end-userdevice, a first map associated with a first topic included in a querysent to an augmented reality platform entity. The example method alsoincludes highlighting areas of interest on the first map. The areas ofinterest associated are associated with second topics. Additionally, theexample method includes presenting a scrubber track, a scrubber, and aplurality of event points. The event points indicate second maps relatedto the first topic. The second maps are associated with different datesthan the first map. In response to receiving, with an input device, aselection of one of the areas of interest, the example method includessending a query to the augmented reality platform entity that includesthe corresponding second topic.

An example method includes presenting, on a display of an end-userdevice, map geometry data. The example method also includes highlightingobjects of interest in the displayed map geometry data. The objects ofinterest are each associated with a topic. In response to receiving,with an input device, a selection of one of the objects of interest, theexample method includes sending a query to an augmented reality platformentity that includes the corresponding topic. Additionally, in responseto receiving a query result from the augmented reality platform entity,the method includes (a) presenting a scrubber track, a scrubber, and aplurality of event points, and (b) displaying a first hierarchy of aplurality of hierarchies associated with records included in the queryresult associated with one of the chronological categories selected by aposition of the scrubber on the scrubber track corresponding to anassociated one of the plurality of event points. The plurality of eventpoints each indicate a chronological category associated with the queryresult.

An example method includes presenting, on a display of an end-userdevice, live image data being captured by a camera of the end-userdevice. The example method also includes highlighting objects ofinterest in the displayed live image data being captured by the camera.The objects of interest are each associated with a topic. In response toreceiving, with an input device, a selection of one of the objects ofinterest, the example method includes sending a query to the augmentedreality platform entity that includes the corresponding topic.Additionally, in response to receiving a query result from the augmentedreality platform entity, the method includes (a) presenting a scrubbertrack, a scrubber, and a plurality of event points, and (b) displaying afirst hierarchy of a plurality of hierarchies associated with recordsincluded in the query result associated with one of the chronologicalcategories selected by a position of the scrubber on the scrubber trackcorresponding to an associated one of the plurality of event points. Theplurality of event points each indicate a chronological categoryassociated with the query result.

An example method includes presenting, on a display of an end-userdevice, live image data being captured by a camera of the end-userdevice. The example method also includes determining a pose of theend-user device. The pose includes a location of the end-user device.Additionally, the example method includes sending the pose of theend-user device to an augmented reality platform entity. The examplemethod includes receiving closed captioning data from the augmentedreality platform entity, the closed captioning data based on thelocation of the end-user device. The close caption data includesindications of first topics within the closed captioning data. Themethod of claim 1 includes superimposing the closed captioning data ontothe live image data.

In one embodiment of the present application, a method includes thesteps of presenting, on a display of an end-user device, live image databeing captured by a camera of the end-user device, determining a pose ofthe end-user device, the pose including a location of the end-userdevice, sending the pose of the end-user device to an augmented realityplatform entity, receiving closed captioning data from the augmentedreality platform entity, the closed captioning data based on thelocation of the end-user device and including an indication of a firsttopic embedded within the closed captioning data, and superimposing theclosed captioning data incorporating an interactive element associatedwith the first topic onto the live image data.

In some embodiments, the method includes the step of highlightingobjects of interest in the displayed live image data being captured bythe camera, the objects of interest associated with a topic. In otherembodiments, the step of highlighting the objects of interest in thedisplayed live image data being captured by the camera includes thesteps of sending the pose of the end-user device to an augmented realityplatform entity, the pose including the location and orientation of theend-user device, and receiving the objects of interest from theaugmented reality platform entity based on the pose of the end-userdevice. In still further embodiments, superimposing the closedcaptioning data onto the live image data includes superimposing theclosed captioning data onto one of the objects of interest. In otherembodiments, the method includes the step of, in response to receiving aselection of one of the objects of interest associated with secondtopics, sending a query to the augmented reality platform entity thatincludes the corresponding one of the second topics.

In another embodiment, the method includes the step of, in response toreceiving a selection of the interactive element within the displayedclosed captioning data, sending a query to the augmented realityplatform entity that includes the corresponding first topic. In someembodiments, the method includes the steps of, in response to receivinga query result from the augmented reality platform entity, presenting ascrubber track, a scrubber, and a plurality of event points, theplurality of event points indicative chronological categories associatedwith the query result, and displaying a first hierarchy of a pluralityof hierarchies associated with records included in the query resultassociated with one of the chronological categories selected by aposition of the scrubber on the scrubber track corresponding to anassociated one of the plurality of event points.

In another embodiment of the present application, an end-user deviceincludes memory with an application and a processor communicativelycoupled to the memory. The application, when executed, causing theprocessor to present, on a display, live image data being captured by acamera of the end-user device, determine a pose of the end-user device,the pose including a location of the end-user device, send the pose ofthe end-user device to an augmented reality platform entity, receiveclosed captioning data from the augmented reality platform entity, theclosed captioning data based on the location of the end-user device andincluding an indication of a first topic embedded within the closedcaptioning data, and superimpose the closed captioning dataincorporating an interactive element associated with the first topiconto the live image data.

In some embodiments, the application, when executed, causes theprocessor to highlight objects of interest in the displayed live imagedata being captured by the camera, the objects of interest associatedwith a topic. In other embodiments, to highlight the objects of interestin the displayed live image data being captured by the camera, theapplication, when executing, causes the processor to send the pose ofthe end-user device to an augmented reality platform entity, the poseincluding the location and orientation of the end-user device, andreceive the objects of interest from the augmented reality platformentity based on the pose of the end-user device. In still furtherembodiments, to superimpose the closed captioning data onto the liveimage data, the application, when executing, causes the processor tosuperimpose the closed captioning data onto one of the objects ofinterest. In another embodiment, the application, when executing, causesthe processor to, in response to receiving a selection of one of theobjects of interest associated with second topics, send a query to theaugmented reality platform entity that includes the corresponding one ofthe second topics.

In yet another example, the application, when executing, causes theprocessor to, in response to receiving a selection of interactiveelement within the displayed closed captioning data, send a query to theaugmented reality platform entity that includes the corresponding firsttopic. In some embodiments, the application, when executing, causes theprocessor to, in response to receiving a query result from the augmentedreality platform entity present a scrubber track, a scrubber, and aplurality of event points, the plurality of event points indicativechronological categories associated with the query result, and display afirst hierarchy of a plurality of hierarchies associated with recordsincluded in the query result associated with one of the chronologicalcategories selected by a position of the scrubber on the scrubber trackcorresponding to an associated one of the plurality of event points.

In a further embodiment of the present application, a tangible computerreadable medium includes instructions that, when executed, cause anend-user device to present, on a display of an end-user device, liveimage data being captured by a camera of the end-user device, determinea pose of the end-user device, the pose including a location of theend-user device, send the pose of the end-user device to an augmentedreality platform entity, receive closed captioning data from theaugmented reality platform entity, the closed captioning data based onthe location of the end-user device and including an indication of afirst topic embedded within the closed captioning data, and superimposethe closed captioning data incorporating an interactive elementassociated with the first topic onto the live image data.

In some embodiments, the instructions, when executed, cause the end-userdevice to highlight objects of interest in the displayed live image databeing captured by the camera, the objects of interest associated with atopic. In other embodiments, to highlight the objects of interest in thedisplayed live image data being captured by the camera, theinstructions, when executed, cause the end-user device to send the poseof the end-user device to an augmented reality platform entity, the poseincluding the location and orientation of the end-user device andreceive the objects of interest from the augmented reality platformentity based on the pose of the end-user device. In still furtherembodiments, superimposing the closed captioning data onto the liveimage data includes superimposing the closed captioning data onto one ofthe objects of interest. In another embodiment, the instructions, whenexecuted, cause the end-user device to, in response to receiving aselection of the interactive element within the closed captioning data,send a query to the augmented reality platform entity that includes thecorresponding first topic. In another embodiment, the instructions, whenexecuted, cause the end-user device to, in response to receiving a queryresult from the augmented reality platform entity present a scrubbertrack, a scrubber, and a plurality of event points, the plurality ofevent points indicative chronological categories associated with thequery result, and display a first hierarchy of a plurality ofhierarchies associated with records included in the query resultassociated with one of the chronological categories selected by aposition of the scrubber on the scrubber track corresponding to anassociated one of the plurality of event points.

In a further embodiment of the present application, a method includesthe steps of: presenting, on a display of an end-user device, a firstmap associated with a first topic included in a query sent to anaugmented reality platform entity; highlighting areas of interest on thefirst map, the areas of interest associated with second topics;presenting a scrubber track, a scrubber, and a plurality of event pointsalong the scrubber track, the event points indicative of second mapsrelated to the first topic associated with different dates; and inresponse to receiving, with an input device, a selection of one of theareas of interest, sending a query to the augmented reality platformentity that includes the corresponding second topic. In someembodiments, including in response to receiving, with the input device,a change in position of the scrubber along the scrubber track to one ofthe plurality of event points, displaying the second map associated withthe corresponding one of the plurality of event points.

In other embodiments, the method includes the steps of receiving searchresults based on the query, the query results includes records organizedinto at least first hierarchies, second hierarchies, and thirdhierarchies, and superimposing an interface over the first map initiallydisplaying the first hierarchies. In a further embodiment, the firsthierarchies are displayed in a first window, and the method includes thesteps of, in response to receiving a selection of one the firsthierarchies, displaying the second hierarchies associated with theselected one the first hierarchies in a second window, and, in responseto receiving a selection of one of the second hierarchies, displayingthe third hierarchies associated with the selected one the secondhierarchies in a third window, the third hierarchies to includeselectable uniform resource locators to retrieve entries associated withthe third hierarchies. In another embodiment, the first hierarchies aredisplayed in a circular window, and the method includes the steps of, inresponse to receiving a selection of one the first hierarchies,displaying the second hierarchies associated with the selected one thefirst hierarchies in a first concentric ring around the circular window,and, in response to receiving a selection of one of the secondhierarchies, displaying the third hierarchies associated with theselected one the second hierarchies in a second concentric ring aroundthe first concentric ring, the third hierarchies to include selectableuniform resource locators to retrieve entries associated with the thirdhierarchies. In a still further embodiment, the plurality of eventpoints is a first plurality of event points, and the method includes thesteps of hiding the first plurality of event points and displaying asecond plurality of event points, the second plurality of event pointsindicative of ones of the search results associated with chronologicalcategories. In another embodiment, the method includes the step of, inresponse to receiving, with the input device, a change in position ofthe scrubber along the scrubber track to one of the second plurality ofevent points, providing the ones of the search results associated withthe corresponding chronological category.

In another embodiment of the present application, an end-user deviceincludes memory with an application and a processor communicativelycoupled to the memory. The application, when executed, causing theprocessor to: present, on a display of the end-user device, a first mapassociated with a first topic included in a query sent to an augmentedreality platform entity; highlight areas of interest on the first map,the areas of interest associated with second topics; present a scrubbertrack, a scrubber, and a plurality of event points along the scrubbertrack, the event points indicative of second maps related to the firsttopic associated with different dates; and, in response to receiving,with an input device, a selection of one of the areas of interest, senda query to the augmented reality platform entity that includes thecorresponding second topic. In some embodiments, the application, whenexecuting, causes the processor to, in response to receiving a change inposition of the scrubber along the scrubber track, with the inputdevice, to one of the plurality of event points, display the second mapassociated with the corresponding one of the plurality of event points.In other embodiments, the application, when executing, causes theprocessor to: receive search results based on the query, the queryresults includes records organized into at least first hierarchies,second hierarchies, and third hierarchies; and superimpose an interfaceover the first map initially displaying the first hierarchies.

In another embodiment, the first hierarchies are displayed in a firstwindow, and the application, when executing, causes the processor to: inresponse to receiving a selection of one the first hierarchies, displaythe second hierarchies associated with the selected one the firsthierarchies in a second window; and in response to receiving a selectionof one of the second hierarchies, display the third hierarchiesassociated with the selected one the second hierarchies in a thirdwindow, the third hierarchies to include selectable uniform resourcelocators to retrieve entries associated with the third hierarchies. In afurther embodiment, the first hierarchies are displayed in a circularwindow, and wherein the application, when executing, causes theprocessor to: in response to receiving a selection of one the firsthierarchies, display the second hierarchies associated with the selectedone the first hierarchies in a first concentric ring around the circularwindow; and in response to receiving a selection of one of the secondhierarchies, display the third hierarchies associated with the selectedone the second hierarchies in a second concentric ring around the firstconcentric ring, the third hierarchies to include selectable uniformresource locators to retrieve entries associated with the thirdhierarchies.

In a still further embodiment, the plurality of event points is a firstplurality of event points, and the application, when executing, causesthe processor to: hide the first plurality of event points; and displaya second plurality of event points, the second plurality of event pointsindicative of ones of the search results associated with chronologicalcategories.

In another embodiment, the application, when executing, causes theprocessor to, in response to receiving with the input device, a changein position of the scrubber along the scrubber track to one of thesecond plurality of event points, providing the ones of the searchresults associated with the corresponding chronological category.

In another embodiment of the present application, a tangible computerreadable medium includes instructions that, when executed, cause anend-user device to: present, on a display, a first map associated with afirst topic included in a query sent to an augmented reality platformentity; highlight areas of interest on the first map, the areas ofinterest associated with second topics; present a scrubber track, ascrubber, and a plurality of event points along the scrubber track, theevent points indicative of second maps related to the first topicassociated with different dates; and in response to receiving, with aninput device, a selection of one of the areas of interest, send a queryto the augmented reality platform entity that includes the correspondingsecond topic. In a further embodiment, the instructions, when executed,cause the end-user device to, in response to receiving, with the inputdevice, a change in position of the scrubber along the scrubber track toone of the plurality of event points, display the second map associatedwith the corresponding one of the plurality of event points. In anotherembodiment, the instructions, when executed, cause the end-user deviceto: receive search results based on the query, the query resultsincludes records organized into at least first hierarchies, secondhierarchies, and third hierarchies; and superimpose an interface overthe first map initially displaying the first hierarchies. In a furtherembodiment, the first hierarchies are displayed in a first window, andwherein the instructions, when executed, cause the end-user device to:in response to receiving a selection of one the first hierarchies,display the second hierarchies associated with the selected one thefirst hierarchies in a second window; and in response to receiving aselection of one of the second hierarchies, display the thirdhierarchies associated with the selected one the second hierarchies in athird window, the third hierarchies to include selectable uniformresource locators to retrieve entries associated with the thirdhierarchies.

In a further embodiment, the first hierarchies are displayed in acircular window, and the instructions, when executed, cause the end-userdevice to, in response to receiving a selection of one the firsthierarchies, display the second hierarchies associated with the selectedone the first hierarchies in a first concentric ring around the circularwindow, and, in response to receiving a selection of one of the secondhierarchies, display the third hierarchies associated with the selectedone the second hierarchies in a second concentric ring around the firstconcentric ring, the third hierarchies to include selectable uniformresource locators to retrieve entries associated with the thirdhierarchies. In another embodiment, the plurality of event points is afirst plurality of event points, and wherein the instructions, whenexecuted, cause the end-user device to: hide the first plurality ofevent points; display a second plurality of event points, the secondplurality of event points indicative of ones of the search resultsassociated with chronological categories; and, in response to receiving,with the input device, a change in position of the scrubber along thescrubber track to one of the second plurality of event points, providethe ones of the search results associated with the correspondingchronological category.

In another embodiment of the present application, a method includes thesteps of: presenting, on a display of an end-user device, map geometrydata; highlighting objects of interest in the displayed map geometrydata, the objects of interest associated with a topic; in response toreceiving, with an input device, a selection of one of the objects ofinterest, sending a query to an augmented reality platform entity thatincludes the corresponding topic. The method further includes the stepsof, in response to receiving a query result from the augmented realityplatform entity, presenting a scrubber track, a scrubber, and aplurality of event points along the scrubber track, the plurality ofevent points indicative chronological categories associated with thequery result; and displaying a first hierarchy of a plurality ofhierarchies associated with records included in the query resultassociated with one of the chronological categories selected by aposition of the scrubber on the scrubber track corresponding to anassociated one of the plurality of event points. In another embodiment,the first hierarchy is displayed in a first window, and the firsthierarchy displays a list of database content providers associated withthe query results with dates within one of the chronological categoriesindicated by the scrubber. In a further embodiment, the method includesthe step of, in response to receiving a selection of one of the databasecontent providers of the first hierarchy, displaying a second hierarchyin a second window, the second hierarchy including a list of topicsassociated with the query results associated with the selected databasecontent providers and associated with within the one of thechronological categories indicated by the scrubber. In a still furtherembodiment, the method includes the step of, in response to receiving aselection of one of the topics of the second hierarchy, displaying athird hierarchy in a third window, the second hierarchy including a listof uniform resource locators of entries associated with the selectedtopic and associated with within the one of the chronological categoriesindicated by the scrubber. In some embodiments, the method includes thestep of, based on the position of the scrubber, superimposing an imageonto the map geometry data over one of the objects of interest, theimage being the one of the objects of interest during the chronologicalcategories indicated by the scrubber. In other embodiments, the methodincludes the step of, for the objects of interest, superimposing aninformation window that includes a portion of search results for thecorresponding topic. In still further embodiments, the method includesthe step of determining which portion of the map geometry data topresent based on the orientation of the end-user device.

In another embodiment of the present application, an end-user deviceincludes memory with an application and a processor communicativelycoupled to the memory. The application, when executed, causing theprocessor to: present, on a display, map geometry data; highlightobjects of interest in the displayed map geometry data, the objects ofinterest associated with a topic; and, in response to receiving, with aninput device, a selection of one of the objects of interest, send aquery to an augmented reality platform entity that includes thecorresponding topic. The method further includes the steps of, inresponse to receiving a query result from the augmented reality platformentity: present a scrubber track, a scrubber, and a plurality of eventpoints along the scrubber track, the plurality of event pointsindicative chronological categories associated with the query result;and display a first hierarchy of a plurality of hierarchies associatedwith records included in the query result associated with one of thechronological categories selected by a position of the scrubber on thescrubber track corresponding to an associated one of the plurality ofevent points.

In another embodiment, the first hierarchy is displayed in a firstwindow, and the first hierarchy displays a list of database contentproviders associated with the query results with dates within one of thechronological categories indicated by the scrubber. In a furtherembodiment, the application, when executing, causes the processor to, inresponse to receiving a selection of one of the database contentproviders of the first hierarchy, display a second hierarchy in a secondwindow, the second hierarchy including a list of topics associated withthe query results associated with the selected database contentproviders and associated with within the one of the chronologicalcategories indicated by the scrubber. In a further embodiment, theapplication, when executing, causes the processor to, in response toreceiving a selection of one of the topics of the second hierarchy,display a third hierarchy in a third window, the second hierarchyincluding a list of uniform resource locators of entries associated withthe selected topic and associated with within the one of thechronological categories indicated by the scrubber.

In some embodiments, the application, when executing, causes theprocessor to, based on the position of the scrubber, superimpose animage onto the map geometry data over one of the objects of interest,the image being the one of the objects of interest during thechronological categories indicated by the scrubber. In otherembodiments, the application, when executing, causes the processor to,for the objects of interest, superimposing an information window thatincludes a portion of search results for the corresponding topic. Instill further embodiments, the application, when executing, causes theprocessor to determining which portion of the map geometry data topresent based on the orientation of the end-user device.

In another embodiment of the present application, a tangible computerreadable medium includes instructions that, when executed, cause anend-user device to: present, on a display, map geometry data; highlightobjects of interest in the displayed map geometry data, the objects ofinterest associated with a topic; and, in response to receiving, with aninput device, a selection of one of the objects of interest, send aquery to an augmented reality platform entity that includes thecorresponding topic. The instructions further cause the end-user deviceto, in response to receiving a query result from the augmented realityplatform entity: present a scrubber track, a scrubber, and a pluralityof event points along the scrubber track, the plurality of event pointsindicative chronological categories associated with the query result;and display a first hierarchy of a plurality of hierarchies associatedwith records included in the query result associated with one of thechronological categories selected by a position of the scrubber on thescrubber track corresponding to an associated one of the plurality ofevent points.

In some embodiments, the first hierarchy is displayed in a first window,and the first hierarchy displays a list of database content providersassociated with the query results with dates within one of thechronological categories indicated by the scrubber. In otherembodiments, the instructions, when executed, cause the end-user deviceto, in response to receiving a selection of one of the database contentproviders of the first hierarchy, display a second hierarchy in a secondwindow, the second hierarchy including a list of topics associated withthe query results associated with the selected database contentproviders and associated with within the one of the chronologicalcategories indicated by the scrubber. In still further embodiments, theinstructions, when executed, cause the end-user device to, in responseto receiving a selection of one of the topics of the second hierarchy,display a third hierarchy in a third window, the second hierarchyincluding a list of uniform resource locators of entries associated withthe selected topic and associated with within the one of thechronological categories indicated by the scrubber. In yet furtherembodiments, the instructions, when executed, cause the end-user deviceto, based on the position of the scrubber, superimposing an image ontothe map geometry data over one of the objects of interest, the imagebeing the one of the objects of interest during the chronologicalcategories indicated by the scrubber. In yet another embodiment, theinstructions, when executed, cause the end-user device to, for theobjects of interest, superimposing an information window that includes aportion of search results for the corresponding topic.

In a further embodiment of the present applications, a method includesthe steps of: presenting, on a display of an end-user device, live imagedata being captured by a camera of the end-user device; highlightingobjects of interest in the displayed live image data being captured bythe camera, the objects of interest associated with a topic; and, inresponse to receiving, with an input device, a selection of one of theobjects of interest, sending a query to the augmented reality platformentity that includes the corresponding topic. The method furtherincludes the steps of, in response to receiving a query result from theaugmented reality platform entity: presenting a scrubber track, ascrubber, and a plurality of event points along the scrubber track, theplurality of event points indicative chronological categories associatedwith the query result; and displaying a first hierarchy of a pluralityof hierarchies associated with records included in the query resultassociated with one of the chronological categories selected by aposition of the scrubber on the scrubber track corresponding to anassociated one of the plurality of event points.

In another embodiment, the step of highlighting the objects of interestin the displayed live image data being captured by the camera includes:determining, with a processor, a pose of the end-user device, the poseincluding a location of the end-user device and an orientation of thecamera of the end-user device; sending the pose of the end-user deviceto an augmented reality platform entity; and receiving the objects ofinterest from the augmented reality platform entity based on the pose ofthe end-user device. In further embodiments, the method includes, forthe objects of interest, superimposing an information window thatincludes a portion of search results for the corresponding topic. In astill further embodiment, the method includes, based on the position ofthe scrubber, superimposing an image onto the map geometry data over oneof the objects of interest, the image being the one of the objects ofinterest during the chronological categories indicated by the scrubber.In a still further embodiment, the first hierarchy is displayed in afirst window, and the first hierarchy displays a list of databasecontent providers associated with the query results with dates withinone of the chronological categories indicated by the scrubber. Inanother embodiment, the method includes, iin response to receiving aselection of one of the database content providers of the firsthierarchy, displaying a second hierarchy in a second window, the secondhierarchy including a list of topics associated with the query resultsassociated with the selected database content providers and associatedwith within the one of the chronological categories indicated by thescrubber. In a still further embodiment, the method includes, inresponse to receiving a selection of one of the topics of the secondhierarchy, displaying a third hierarchy in a third window, the secondhierarchy including a list of uniform resource locators of entriesassociated with the selected topic and associated with within the one ofthe chronological categories indicated by the scrubber.

In another embodiment of the present application, an end-user deviceincludes memory with an application and a processor communicativelycoupled to the memory. The application, when executed, causing theprocessor to: present, on a display, live image data being captured by acamera of the end-user device; highlight objects of interest in thedisplayed live image data being captured by the camera, the objects ofinterest associated with a topic; and in response to receiving, with aninput device, a selection of one of the objects of interest, send aquery to the augmented reality platform entity that includes thecorresponding topic. The application further causes the processor to, inresponse to receiving a query result from the augmented reality platformentity: present a scrubber track, a scrubber, and a plurality of eventpoints along the scrubber track, the plurality of event pointsindicative chronological categories associated with the query result;and display a first hierarchy of a plurality of hierarchies associatedwith records included in the query result associated with one of thechronological categories selected by a position of the scrubber on thescrubber track corresponding to an associated one of the plurality ofevent points.

In another embodiment, to highlight the objects of interest in thedisplayed live image data being captured by the camera, the application,when executing, causes the processor to: determine, with a processor, apose of the end-user device, the pose including a location of theend-user device and an orientation of the camera of the end-user device;send the pose of the end-user device to an augmented reality platformentity; receive the objects of interest from the augmented realityplatform entity based on the pose of the end-user device. In a furtherembodiment, the application, when executing, causes the processor to,for the objects of interest, superimpose an information window thatincludes a portion of search results for the corresponding topic. Inanother embodiment, the application, when executing, causes theprocessor to, based on the position of the scrubber, superimposing animage onto the map geometry data over one of the objects of interest,the image being the one of the objects of interest during thechronological categories indicated by the scrubber. In a still furtherembodiment, the first hierarchy is displayed in a first window, and thefirst hierarchy displays a list of database content providers associatedwith the query results with dates within one of the chronologicalcategories indicated by the scrubber. In yet another embodiment, theapplication, when executing, causes the processor to, in response toreceiving a selection of one of the database content providers of thefirst hierarchy, display a second hierarchy in a second window, thesecond hierarchy including a list of topics associated with the queryresults associated with the selected database content providers andassociated with within the one of the chronological categories indicatedby the scrubber. In another embodiment, the application, when executing,causes the processor to, in response to receiving a selection of one ofthe topics of the second hierarchy, displaying a third hierarchy in athird window, the second hierarchy including a list of uniform resourcelocators of entries associated with the selected topic and associatedwith within the one of the chronological categories indicated by thescrubber.

In another embodiment of the present application, a tangible computerreadable medium comprising instructions that, when executed, cause anend-user device to: present, on a display of an end-user device, liveimage data being captured by a camera of the end-user device; highlightobjects of interest in the displayed live image data being captured bythe camera, the objects of interest associated with a topic; and, inresponse to receiving, with an input device, a selection of one of theobjects of interest, send a query to the augmented reality platformentity that includes the corresponding topic. The instructions furthercause the end-user device to, in response to receiving a query resultfrom the augmented reality platform entity: present a scrubber track, ascrubber, and a plurality of event points along the scrubber track, theplurality of event points indicative chronological categories associatedwith the query result; and display a first hierarchy of a plurality ofhierarchies associated with records included in the query resultassociated with one of the chronological categories selected by aposition of the scrubber on the scrubber track corresponding to anassociated one of the plurality of event points.

In another embodiment, to highlight the objects of interest in thedisplayed live image data being captured by the camera, theinstructions, when executed, cause the end-user device to: determine,with a processor, a pose of the end-user device, the pose including alocation of the end-user device and an orientation of the camera of theend-user device; send the pose of the end-user device to an augmentedreality platform entity; and receive the objects of interest from theaugmented reality platform entity based on the pose of the end-userdevice. In another embodiment, the instructions, when executed, causethe end-user device to, based on the position of the scrubber,superimpose an image onto the map geometry data over one of the objectsof interest, the image being the one of the objects of interest duringthe chronological categories indicated by the scrubber. In a stillfurther embodiment, the first hierarchy is displayed in a first window,and the first hierarchy displays a list of database content providersassociated with the query results with dates within one of thechronological categories indicated by the scrubber. In anotherembodiment, the instructions, when executed, cause the end-user deviceto, in response to receiving a selection of one of the database contentproviders of the first hierarchy, display a second hierarchy in a secondwindow, the second hierarchy including a list of topics associated withthe query results associated with the selected database contentproviders and associated with within the one of the chronologicalcategories indicated by the scrubber. In a still further embodiment, theinstructions, when executed, cause the end-user device to, in responseto receiving a selection of one of the topics of the second hierarchy,display a third hierarchy in a third window, the second hierarchyincluding a list of uniform resource locators of entries associated withthe selected topic and associated with within the one of thechronological categories indicated by the scrubber.

In another embodiment of the present application, a method includes thesteps of: generating, with a processor, records for entries from aplurality of database content providers, the records identifying topicsincluded in the entries and dates associated with the entries; andanalyzing the entries from the plurality of database content providersto determine affinities between the topics. The method further includesthe steps of, in response to receiving a query that includes a searchtopic from an end-user device: retrieving, from a database, recordsidentifying the search topic and related topics that have affinity withthe search topic; organizing, with the processor, the records intochronological categories based on the dates associated with thecorresponding entries, each one of the chronological categoriesrepresenting a different period of time, the chronological categoriesbeing based on a number of the dates associated with the retrievedrecords and a clustering of dates associated with the retrieved records;for each of the chronological categories, generating a query result withthe processor, the query result organizing the records into hierarchiesbased on attributes of the entries associated with the records; andsending the query result to the end-user device. In another embodiment,the hierarchies include at least first hierarchies, second hierarchies,and third hierarchies.

In a further embodiment, organizing the query result into thehierarchies includes: organizing the records into the first hierarchiesbased on which of the plurality of database content providers isassociated with the records; for each of the plurality of databasecontent providers in the first hierarchies, organizing the records intothe second hierarchies based on the corresponding topics; and, for eachof the topics in the second hierarchies, organizing the records into thethird hierarchies based on a category of the entries associated with therecords. In a still further embodiment, generating the records for theentries from the plurality of database content providers includes, foreach of the entries, determining whether the entry is an image; and, inresponse to the entry being an image, determining the topics associatedwith the image based on at least one of metadata included in the image,image recognition performed on the image, or articles in which the imageis used. In another embodiment, each of the chronological categories areassociated with an event mark when the query results are displayed onthe end-user device. In another embodiment, the method includes the stepof comparing words in the entries match topic records in a topicdatabase to identify which of the topics are includes in the entries. Ina further embodiment, the method includes the step of analyzing theentries from the plurality of database content providers to determinepotential topics to be added to the topic database.

In a further embodiment of the present application, a system includesmemory having instructions and a processor communicatively coupled tothe memory. The instructions, when executed, causing the processor to:generate records for entries from a plurality of database contentproviders, the records identifying topics included in the entries anddates associated with the entries; and analyze the entries from theplurality of database content providers to determine affinities betweenthe topics. The instructions further cause the processor to, in responseto receiving a query that includes a search topic from an end-userdevice: retrieve, from a database, records identifying the search topicand related topics that have affinity with the search topic; organizethe records into chronological categories based on the dates associatedwith the corresponding entries, each one of the chronological categoriesrepresenting a different time period, the time periods associated withthe chronological categories being based on a number of the datesassociated with the retrieved records and a clustering of datesassociated with the retrieved records; for each of the chronologicalcategories, generate a query result with the processor, the query resultorganizing the records into hierarchies based on attributes of theentries associated with the records; and send the query result to theend-user device. In another embodiment, the hierarchies include at leastfirst hierarchies, second hierarchies, and third hierarchies. In afurther embodiment, to organize the query result into the hierarchies,the processor is to: organize the records into the first hierarchiesbased on which of the plurality of database content providers isassociated with the records; for each of the plurality of databasecontent providers in the first hierarchies, organize the records intothe second hierarchies based on the corresponding topics; and for eachof the topics in the second hierarchies, organize the records into thethird hierarchies based on a category of the entries associated with therecords. In yet another embodiment, where to generate the records forthe entries from the plurality of database content providers, theprocessor is to, for each of the entries: determine whether the entry isan image; and in response to the entry being an image, determine thetopics associated with the image based on at least one of metadataincluded in the image, image recognition performed on the image, orarticles in which the image is used. In a still further embodiment, thechronological categories are each associated with an event marker whenthe query results are displayed on the end-user device. In anotherembodiment, the processor is to compare words in the entries match topicrecords in a topic database to identify which of the topics are includesin the entries. In a further embodiment, the processor is to analyze theentries from the plurality of database content providers to determinepotential topics to be added to the topic database.

In another embodiment of the present application, a tangible computerreadable medium includes instructions that, when executed, cause aserver to: generate records for entries from a plurality of databasecontent providers, the records identifying topics included in theentries and dates associated with the entries; and analyze the entriesfrom the plurality of database content providers to determine affinitiesbetween the topics. The instructions further cause the server to, inresponse to receiving a query that includes a search topic from anend-user device: retrieve, from a database, records identifying thesearch topic and related topics that have affinity with the searchtopic; organize the records into chronological categories based on thedates associated with the corresponding entries, each one of thechronological categories representing a different time period, the timeperiods associated with the chronological categories being based on anumber of the dates associated with the retrieved records and aclustering of dates associated with the retrieved records; for each ofthe chronological categories, generate a query result with theprocessor, the query result organizing the records into hierarchiesbased on attributes of the entries associated with the records; and sendthe query result to the end-user device. In another embodiment, thehierarchies include at least first hierarchies, second hierarchies, andthird hierarchies. In yet another embodiment, the instructions, whenexecuted, cause the server to: organize the records into the firsthierarchies based on which of the plurality of database contentproviders is associated with the records; for each of the plurality ofdatabase content providers in the first hierarchies, organize therecords into the second hierarchies based on the corresponding topics;and for each of the topics in the second hierarchies, organize therecords into the third hierarchies based on a category of the entriesassociated with the records. In a still further embodiment, to generatethe records for the entries from the plurality of database contentproviders, the instructions cause the server to, for each of theentries: determine whether the entry is an image; and, in response tothe entry being an image, determine the topics associated with the imagebased on at least one of metadata included in the image, imagerecognition performed on the image, or articles in which the image isused. In yet another embodiment, the chronological categories areassociated with event markers when the query results are displayed onthe end-user device. In a further embodiment, the instructions, whenexecuted cause the server to: compare words in the entries match topicrecords in a topic database to identify which of the topics are includesin the entries; and analyze the entries from the plurality of databasecontent providers to determine potential topics to be added to the topicdatabase.

In a still further embodiment of the present application, a methodincludes the steps of: presenting, on a display of an end-user device,map geometry data; presenting a scrubber track, a position marker, and aplurality of event points along the scrubber track, the plurality ofevent points indicative chronological categories; and displaying indiciain the displayed map geometry data when one of the chronologicalcategories corresponding to a timestamp associated with the indicia isselected by a position of the position marker on the scrubber trackcorresponding to an associated one of the plurality of event points, theindicia being each associated with an event; in response to receiving,with an input device, a selection of one of the indicia, sending a queryto an augmented reality platform entity that includes topic related thecorresponding event; and displaying a first hierarchy of a plurality ofhierarchies associated with records included in a query result receivedfrom the augmented reality platform entity relating to the selectedindicia.

In a further embodiment of the present application, a method includesthe steps of: presenting, on a display of an end-user device, live imagedata being captured by a camera of the end-user device; determining apose of the end-user device, the pose including a location of theend-user device; sending the pose of the end-user device to an augmentedreality platform entity; receiving tracking data for athletes in theimage data based on the pose of the end-user device; superimposing thetracking data on the locations of the athletes in the live image databeing captured by a camera of the end-user device; in response toreceiving a selection of the superimposed tracking data of one of theathletes, sending a query to the augmented reality platform entity thatincludes an identity of the selected athlete; and displaying a firsthierarchy of a plurality of hierarchies associated with records includedin a query result received from the augmented reality platform entityrelating to the selected athlete.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the invention, reference may be made toembodiments shown in the following drawings. The components in thedrawings are not necessarily to scale and related elements may beomitted, or in some instances proportions may have been exaggerated, soas to emphasize and clearly illustrate the novel features describedherein. In addition, system components can be variously arranged, asknown in the art. Further, in the drawings, like reference numeralsdesignate corresponding parts throughout the several views.

FIG. 1 illustrates a system diagram of an augmented reality platformentity providing query results in response to receiving queries fromend-user devices in accordance to the teachings of this disclosure.

FIG. 2 illustrates an example data structure that may be used to storethe entry records in the augmented reality platform database.

FIG. 3 illustrates an example data structure that may be used to formthe query.

FIG. 4 illustrates an example data structure used to form the queryresults.

FIG. 5 illustrates a diagram of a navigation map for a user to searchcontent with the end-user device of FIG. 1 .

FIG. 6 depicts an interface presented by the end-user device of FIG. 1to facilitate presenting and interacting with the query results.

FIG. 7 depicts another interface presented by the end-user device ofFIG. 1 to facilitate presenting and interacting with the query results.

FIG. 8 depicts another interface presented by the end-user device ofFIG. 1 to facilitate presenting and interacting with the query results.

FIG. 9 depicts a map geometry interface presented by the end-user deviceof FIG. 1 to facilitate generating the query and presenting andinteracting with the query results.

FIG. 10 depicts a map interface presented by the end-user device of FIG.1 to facilitate generating the query and presenting and interacting withthe query results.

FIG. 11 depicts an augmented reality interface presented by the end-userdevice of FIG. 1 to facilitate generating the query and presenting andinteracting with the query results.

FIG. 12 is a block diagram of the database sorter of FIG. 1 thatgenerates entry records.

FIG. 13 is a block diagram of a query responder of the augmented realityplatform entity of FIG. 1 .

FIG. 14 is a block diagram of a query generator of the augmented realityplatform handler of FIG. 11 .

FIG. 15 is a flowchart of a method to index entries from contentdatabase providers, which may be implemented by the processor platformof FIG. 25 .

FIG. 16 is a flowchart of a method to generate the queries and presentthe query results, which may be implemented by the electronic componentsof FIG. 26 .

FIG. 17 is a flowchart of a method to present the query results, whichmay be implemented by the electronic components of FIG. 26 .

FIG. 18 is a flowchart of a method to present the query results, whichmay be implemented by the electronic components of FIG. 26 .

FIG. 19 is a flowchart of a method to present the query results, whichmay be implemented by the electronic components of FIG. 26 .

FIG. 20 is a flowchart of a method to present the query results inconjunction with map geometry data, which may be implemented by theelectronic components of FIG. 26 .

FIG. 21 a flowchart of a method to present the query results inconjunction with map data, which may be implemented by the electroniccomponents of FIG. 26 .

FIG. 22 a flowchart of a method to present the query results inconjunction with live image data from a camera, which may be implementedby the electronic components of FIG. 26 .

FIG. 23 illustrates the augmented reality interface of FIG. 11 providinglocation specific content superimposed on live images captured by theend-user device of FIG. 1 .

FIG. 24 is a flowchart of a method to detect and provide locationspecific content superimposed on live images captured by the end-userdevice of FIG. 1 , which may be implemented by the electronic componentsof FIG. 26 .

FIG. 25 is a block diagram of a processor platform that may implementthe method of FIG. 15 .

FIG. 26 is a block diagram of electronic components that may implementthe methods of FIGS. 16-22, and 24 .

FIG. 27 depicts a law enforcement map interface presented by theend-user device of FIG. 1 to facilitate generating the query andpresenting and interacting with the query results related to lawenforcement and emergency services.

FIG. 28 illustrates the augmented reality interface of FIG. 11 providinglocation specific sporting event content superimposed on live imagescaptured by the end-user device of FIG. 1 .

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

While the invention may be embodied in various forms, there are shown inthe drawings, and will hereinafter be described, some exemplary andnon-limiting embodiments, with the understanding that the presentdisclosure is to be considered an exemplification of the invention andis not intended to limit the invention to the specific embodimentsillustrated.

As end-user devices (e.g., computers, gaming consoles, smart phones,tablets, smart televisions, over-the-top devices, optical head-mounteddisplays, augmented reality smart glasses, virtual reality headsets,cockpit displays, virtual cockpits, etc.) are developed to include moreprocessing power and better input devices, users expect a more immersiveexperience that facilitates utilizing the environment around them forentertainment, richer content (e.g., context aware content) and moreuseful information. As disclosed below, an augmented reality platformentity provides an interface to facilitate a user, via an end-userdevice, searching for information based on contextual information inimages, maps, map geometry data, and/or live video (e.g., via augmentedreality, virtual reality, etc.). As used herein, “augmented reality”refers to a live video of an environment captured by a camera thatincludes elements superimposed on the live view includingcomputer-generated sounds, images, and/or videos, etc. As used herein,“map geometry data” refers to spatial data on an area that facilitates(i) identifying of points of interest (e.g., buildings, infrastructure,landmarks, etc.) within an associated image and/or (b) determining thelocation and orientation of points of interest with reference to thelocation of the end-user device and orientation of a camera of theend-user device. The map geometry data provides information so that theend-user device is virtually at the location viewed on the en-userdevice. The map geometry data may include representations of locationsthat do not presently exist. For example, the map geometry data mayinclude graphical representations of structures that do not currentlyexist, but may be planned for the future or existed in the past.

The augmented reality platform entity indexes entries from contentdatabase providers. Additionally, in some examples, the augmentedreality platform entity facilitates creation of entries by end-users andindexes those entries. As used herein, “entries” are discrete pieces ofcontent (e.g., articles, images, websites, videos, advertisements,records, map geometry, social media information, etc.) stored by thecontent database providers and/or created by end-users. The contentdatabase providers store and provide various types of entries. Forexample, content database providers include publishers (e.g.,newspapers, magazines, electronic book depositories, blogs, mediaoutlets, etc.), search engines (e.g., Google®, Bing®, Baidu®, etc.),public information offices (e.g., property records, criminal records,incorporation records, licensing records, etc.), research providers(e.g., LexisNexis®, Westlaw®, Bloomberg®, etc.), social media companies(e.g. Facebook®, Twitter®, Weibo®, etc.) map geometry providers (e.g.,Google®, Apple®, Tencent®, etc.), institutional providers (e.g.,libraries, museums, private collections, etc.) education providers(e.g., universities, community colleges, secondary schools, etc.),advertisers, entertainment (e.g., Netflix®, Hulu®, etc.), and/oraudiovisual providers (e.g., Youtube®, Vimeo®, Spotify®, etc.). Somecontent database providers provide public access (e.g., via the WorldWide Web, etc.) to the entries. Other content database providers provideaccess to authorized users (e.g., via login credentials, via keys, etc.)Some content providers provide access via an Application ProgramInterface (API). In some examples, the augmented reality platform entitysearches for and indexes the databases via network and Internetprotocols (e.g., HTML, FTP, etc.).

The augmented reality platform entity analyzes the entries from thecontent database providers to identify topics (e.g., a person, place,thing, or event) within the entries. For example, if an entry beinganalyzed is a real estate listing from a real estate database, theaugmented reality platform entity may identify the street of theproperty, the municipality in which the property is located, previousowners of the property, crime statistics, and/or county recorderrecords, etc. The augmented reality platform entity generates tags forthe entries that uniquely identify the topics associated with theentries. The augmented reality platform entity then creates records forthe entries with the tags, metadata associated with the entries (e.g.,title, geographical coordinates, dates associated with the entry, etc.),and/or a Uniform Resource Identifier (URI) that identifies the locationon a network at which the entry is accessible. In some examples, the URIis a Uniform Resource Location (URL) that provides a protocol identifier(e.g., http, https, ftp, etc.) and an identifier of the on a network(e.g., an intranet, the Internet, etc.). Additionally, through analyzingthe entries from the various content database providers, the augmentedreality platform entity determines affinities (e.g., co-occurrencerelationships) between topics. For example, the topic of the “RookeryBuilding” may be connected to the topics of “Daniel Burnham,” “FrankLloyd Wright,” “Roman Revival Architecture,” and “Burnham & Root,” etc.

After receiving a query from an end-user device, the augmented realityplatform entity determines which records are responsive to the querybased on the topic(s) contained in the query and other topics related(e.g., have affinity and are relevant, etc.) to those topics. Theaugmented reality platform entity generates a response that includesmetadata (e.g., title, author, size, etc.) and the URI associated withthe responsive records. In some examples, the augmented reality platformentity receives pre-query data from the end-user device. In suchexamples, the pre-query data includes map geometry data, image date,device pose data (e.g., location and orientation), and/or image depthmapping (e.g., from a LiDAR and/RADAR on the device). In such examples,the augmented reality platform entity identifies (e.g. via imagerecognition, via coordinate and orientation analysis, image depthmapping analysis, etc.) topics in the pre-query data that the user mayselect to form a query. For example, based on the pre-query data (thecoordinates of a phone, orientation data regarding the camera of thephone, and field of view data of the camera of the phone, etc.) and mapgeometry data from one of the map geometry database providers, theaugmented reality platform entity may determine that the image in thepre-query data includes the Rookery Building and the Chicago Board ofTrade building.

The end-user devices display an interface to facilitate users (i)submitting queries, (ii) browsing the responses to the queries, and(iii) selecting and viewing the content associated with the responses.The end-user device includes visual, audio, and/or haptic outputs. Thevisual outputs may include a screen (e.g., a screen on a phone, atablet, a virtual reality headset, a virtual cockpit, etc.) or aprojection onto a transparent surface (e.g., the panels of an opticalhead-mounted display, augmented reality smart glasses, a heads updisplay, a cockpit glass display, etc.). The interface includes ascrubber track and a scrubber that controls the results that arecurrently display on the end-user device. In some examples, the resultsare listed in chronological order and the scrubber track and thescrubber facilitates scrolling through the results chronologically. Insuch examples, the scrubber track includes event points indicative ofresults corresponding to a particular periods of time (e.g., weeks,months, years, decades, etc.). When the scrubber is moved to aparticular event point, the interface displays results associated withcorresponding period of time. For example, if the scrubber is moved toan event point associated with the year 1893, the interface may displayresults associated the year 1893 (e.g., articles written in 1893, imagesfrom 1893, videos about events in 1893, articles about people alive in1983, etc.) In some examples, the results are listed based on relevancescore and the scrubber track and the scrubber facilitates scrollingthrough the results based on relevance to the query.

The end-user devices include inputs to facilitate (a) manually enteringa query and browsing results (e.g., a touch screen, a mouse, acontroller, microphones, etc.), tracking a user (e.g., a finger, eyegaze location, etc.) to enter queries and browse results (e.g., aforward-facing camera, back-facing cameras, etc), and/or (c) gatheringdata for the interface to identify and/or suggest queries (e.g.,back-facing cameras, global positioning system (GPS) receivers,accelerometers, etc.). In some examples, based on the inputs, theend-user device sends pre-query data to the augmented reality platformentity for the augmented reality platform entity to identify and/orsuggest topics for queries. For example, the end-user device may send animage of the Cloud Gate sculpture, the coordinated of the end-userdevice (e.g., 41.882772 and −87.622958), and the orientation of theend-user device (e.g., 225 degrees from true north). In such an example,the augmented reality platform entity may identify the Cloud Gatesculpture as a suggested topic. In such an example, the end-user deviceshighlights (e.g., superimposes a semi-transparent image) the Cloud Gatesculpture as it is being captured by the camera. In such an example, theend-user device sends a query with the Cloud Gate sculpture as a topicin response to the user (e.g., via the touch screen) indicating interestin the highlighted Cloud Gate sculpture.

FIG. 1 illustrates a system diagram of an augmented reality platformentity (ARPE) 100 providing query results 102 in response to receivingqueries 104 from end-user devices 106 in accordance to the teachings ofthis disclosure. The example ARPE 100, the example end-user devices 106,and example content database providers 108 are communicatively coupledvia a network (e.g., an intranet, the Internet, etc.). The exampleend-user devices 106 include computers (e.g., Window®-based computers,iOS®-based computers, Linux-based computers, etc.), smart phones (e.g.,Android™-based phones, iOS-based phones, Windows® Mobile-based phones,etc.), tablets (e.g., iPad®, Google Pixel®, etc.), smart televisions,over-the-top devices (e.g., Apple TV®, Chromecast™, Amazon Fire®, etc.)optical head-mounted displays (e.g. Google Glass, Vuzix M300, etc.),augmented reality smart glasses (e.g., Microsoft® HoloLens, etc.),virtual reality headsets (e.g., Oculus Rift, HTC Vive, etc.), vehiclecenter console displays, heads up displays, virtual cockpits, glasscockpits, and/or global positioning system (GPS) devices, etc. Asdisclosed in more detail below in FIGS. 14 and 15 below, the end-userdevices 106 present the query results 102 to a user and receive inputfrom the user to form the queries 104 and navigate the entriesassociated with the query results 102.

The content database providers 108 generate and/or store content (e.g.,articles, websites, images, records, maps, map geometry data, videos,etc.) that is accessible via a Uniform Resource Indicator (URI), such asa Uniform Resource Locator (URL). Some content database providers 108provide open access (e.g., without credentials), while other contentdatabase providers 108 provide access to the corresponding contentthrough a login. In the illustrated example, the content databaseproviders 108 include content database providers include publishers(e.g., newspapers, magazines, electronic book depositories, blogs, mediaoutlets, broadcasters (live and pre-taped), etc.), search engines (e.g.,Google®, Bing®, Baidu®, etc.), public information offices (e.g.,property records, criminal records, incorporation records, licensingrecords, etc.), research providers (e.g., LexisNexis®, Westlaw®,Bloomberg®, etc.), social media companies (e.g. Facebook®, Twitter®,Weibo®, etc.) map geometry providers (e.g., Google®, Apple®, Tencent®,etc.), institutional providers (e.g., libraries, museums, privatecollections, etc.) education providers (e.g., universities, communitycolleges, secondary schools, etc.), advertisers, entertainment (e.g.,Netflix®, Hulu®, etc.), and/or audiovisual providers (e.g., Youtube®,Vimeo®, Spotify®, Twitch®, etc.).

In the illustrated example, the ARPE 100 includes a database sorter 110,an augmented reality platform (ARP) database 112, a user handler 114, auser database 116, a session handler 118, and a query responder 120. Asdisclosed in more detail in FIG. 9 below, the example database sorter110 analyzed entries from content database providers 108. The entriesare discrete pieces of content that are associated with a URI that maybe accessed by the end-user device 106 via the URI. For an entry, thedatabase sorter 110 determines which topic(s) are relevant to the entry,and creates an entry record to store in the ARP database 112 thatinclude the identified topic(s), metadata that identifies the entry(e.g., a title, a thumbnail, an author) and the associated contentdatabase provide 108, and the URI to access the entry. In some examples,the database sorter 110 also assigns one or more dates to the entryrecord. For example, an entry about the great Chicago fire may beassigned a dates of Oct. 8, 1871, Oct. 9, 1871, and Oct. 10, 1871.Additionally, in some examples, the database sorter 110 determinesaffinities between the topics. An affinity is a measure of a likelihoodof the co-occurrence of two or more topics in an entry. Topics with highaffinity may be considered related. For example, the topic of “S. R.Crown Hall” may have a high affinity with the topic of “Ludwig Mies vander Rohe.” That is, if a query 104 includes the topic of “S. R. CrownHall,” the user may be also interested in the topic of “Ludwig Mies vander Rohe.”

FIG. 2 illustrates an example data structure 200 that may be used tostore the entry records (e.g. the entry records 1200 of FIG. 12 below)in the ARP database 112. In the illustrated example, the data structure200 includes a record identifier (ID) field 202, a content databaseprovider (CDP) ID field 204, a topic ID field 206, a topic metadatafield 208, topic date field 210, and a URI field 212. The example recordID field 202 includes a numeric or an alphanumeric value that uniquelyidentifies the entry record. The example CDP ID field 204 includes anumeric, an alphanumeric, or an alphabetic value that uniquelyidentifies the content database provider 108 from which the entry recordwas created. The example topic ID field 206 includes a numeric, analphanumeric, or an alphabetic value that uniquely identifies thetopic(s) assigned to the entry record. The topic metadata field 208includes a set of values to be used to inform the user about the contentat the associated URI (e.g., as identified by the URI field 212, etc.).For example, the topic metadata field 208 may include a title, anauthor, a source name, and/or a download size, etc. The topic date field210 identifies dates associated with the entry. The URI field 212includes the URI at which the entry may be found. For example, the URIfield 212 may include a URL, an international standard book number(ISBN), a universal product code (UPC), and/or a global trade itemnumber (GTIN), etc. In such an example where the URI field 212 includesan object identifier (e.g., an ISBN, a UPC, a GTIN, etc.) instead of aURL, the end-user device 106 may use a preferred content databaseprovider 108 (e.g., Amazon.com, the Library of Congress, etc.) when theentry associated with the object identifier is selected by the user.

Returning to FIG. 1 , the user handler 114 maintains the user database116. The when a user first registers with the ARPE 100 (e.g., via theend-user device 106), the user handler 114 manages a registrationprocess to create user records to store the user database 116. The userhandler 114 manages authentication (e.g., via login credentials, etc.)of users accessing the ARPE 100. Additionally, the user handler 114manages and updates the user records to reflect settings and preferencesof the corresponding user. Example settings and preferences may includea preferred query result interface (e.g., the interfaces disclosed inFIGS. 3, 4, and 5 below), bookmarks, preferred content databaseproviders, and/or subscriptions and credentials to content databaseproviders, etc.

The example session handler 118 manages connections and communicationwith the end-user devices 106. The end-user devices 106 communicativelycouple to the session handler 118 via the network. The network may be apublic network, such as the Internet; a private network, such as anintranet; or combinations thereof, and may utilize a variety ofnetworking protocols now available or later developed including, but notlimited to, TCP/IP-based networking protocols. The example sessionhandler 118 receives the queries 104, pre-query data (e.g., informationthat provides context to the location and environment of the end-userdevice 106), registration data, login data, and/or preference data, etc.from the end-user devices 106. The session handler 118 sends sessiondata (e.g., session identifiers, authentication data, etc.), pre-queryresponses, and query results 102.

FIG. 3 illustrates an example data structure 300 that may be used toform the query 104. In the illustrated example, the data structure 300includes an example session ID field 302, an example user ID field 304,an example query data field 306, and an example device pose data field308. The session ID field 302 includes a numeric, an alphanumeric, or analphabetic value that uniquely identifies a session associated with theend-user device 106 that send the query 104. The example user ID field304 includes a numeric, an alphanumeric, or an alphabetic value thatuniquely identifies a user of the end-user device 106. Preferences andsettings may be obtained from the user database 116 with the value inthe user ID field 304. The query data field 306 includes data (e.g.,string data, image data, coordinate data, etc.) to be used by the queryresponder 120 to generate the query results 102. The device pose datafield 308 includes information regarding the pose (e.g., the location,orientation, altitude, etc.) to the end-user device 106.

As disclosed in more detail in FIG. 13 below, the example queryresponder 120 generates query results 102 in response to the queries 104from the end-user devices 106. To generate a query result 102, the queryresponder 120 analyzes the topic(s) included in the query 104 anddetermines which of the query records in the ARP database 112 (a) areresponsive (e.g., are about the topic(s) in the query 104 and are abouttopics that have a high affinity to the topic(s) in the query 104), and(b) conform to the user's preferences. For example, in response to aquery 104 that include the topic of “The Rookery Building,” the queryresponder 120 may generate the query result 102 based on entry recordsindexed for topics such as “The Rookery Building,” “Daniel Burnham,”“Brooks Brothers,” and “South La Salle Street,” etc., and exclude entryrecords associated with content database providers that require asubscription.

FIG. 4 illustrates an example data structure 400 used to form the queryresults 102. In the illustrated example, the query results are arrangedin a hierarchy, based on the CDP ID field 204, the topic ID field 206,and then the entry record data (e.g., the topic metadata field 208, thetopic date field 210, and the URI field 212). However, the datastructure 400 may be arranged in any other suitable manner (e.g., by thetopic ID field 206 then the CDP ID field 204, by the topic date field210, etc.). Alternatively, in some examples, the data structure 400 maybe a flat list of entry record data structures 200 that is thenorganized into a hierarchy by the end-user device 106.

FIG. 5 illustrates a diagram of a navigation map 500 for a user tosearch content with the end-user device 106 of FIG. 1 . In theillustrated example, the navigation map 500 includes a registrationinterface 502, a login interface 504, a home interface 506, a customupdates interface 508, a bookmarks interface 510, and content navigationinterfaces 512 to be displayed on a viewport 514. The registrationinterface 502 facilitates a user creating an account to choose settings,set preferences, and create authentication credentials. Through theregistration interface 502, the user handler 114 creates a user recordto be sorted in the user database 116. The login interface 504 creates asession by facilitating a registered user entering authenticationcredentials. The home interface 506 facilitates selecting one of thecontent navigation interfaces 512, the custom update interface 508and/or the bookmarks interface 510. In some examples, the home interface506 includes one of the other interfaces 508-512 by default depending onpreference settings of the corresponding user record. The custom updateinterface 508 presents entries that are updated in real-time based onpreferences set in the corresponding user record. The bookmarksinterface 510 presents saved previous query results 102.

In the illustrated example, the content navigation interfaces 512includes a text-based query interface 516, a map geometry queryinterface 518, a map query interface 520, and an augmented reality queryinterface 522. The text-based query interface 516 facilitates a usergenerating a query 104 by entering a string into a text box. Forexample, a user may enter “The Thompson Center” into the text box of thetext-based query interface 516. As disclosed in more detail in FIG. 9below, the map geometry query interface 518 provides an interface to (i)generate queries based on interactive map geometry data (e.g., from amap geometry content database provider, such as Google, CycloMedia,Tencent, etc.) and/or static images, and (ii) browse results from thequeries. As disclosed in more detail in FIG. 10 , the map queryinterface 520 provides an interface to (i) generate queries based on mapimages, and (ii) browse results from the queries. As disclosed in moredetail below in FIG. 11 , the augmented reality query interface 522provides an interface to (i) generate queries based on pointing a camera(e.g., the back-facing camera 2616 of FIG. 26 below) at objects, (ii)provide quick results based on identifying the objects, and (iii) browseresults from the queries.

The viewport 514 is the visible area of the interfaces 502-512 on theend-user devices 106. Because the displays (e.g., the display 2612 ofFIG. 26 below), and thus the amount of area to provide the interface tothe user, of the different types of end-user devices 106 have differentdimensions, the viewport 514 varies how the interfaces are displayed onthe particular end-user device 106. For example, the displayedinterfaces 502-512 may be smaller, oriented differently, and/or have adifferent aspect ratio on a portable device (e.g., a smart phone, atablet, etc.) than on a computer screen. In the illustrated example, theviewport 514 formats the orientation, layout, and size to the interfaces502-512 suitable for the particular display of the end-user device 106.

FIG. 6 depicts an interface 600 presented by the end-user device 106 ofFIG. 1 to facilitate presenting and interacting with the query results102. In some examples, the interface 600 is superimposed on theinterface 516-522 from which the query 104 was generated. For example,if the query 104 was generated via the map geometry query interface 518,the interface 600 maybe superimposed on the image of the map geometrydata of the map geometry query interface 518. In the illustratedexample, the interface 600 includes a query identifier field 602, queryresult fields 604, a scrubber track 606, event points 608, timelinemarkers 610, a scrubber 612, and a position marker 614. The examplequery identifier field 602 displays the query 104 for which the queryresults 102 are being displayed. The query result fields 604 display thequery results 102 segregated by topic and/or content database provider108. In some examples, the query result fields 604 are sized accordingto a number of entries associated with the corresponding topic and/orcontent database provider 108. In some examples, when there are moreentries for a topic and/or content database provider 108 than can bedisplayed in the corresponding query result field 604, the interface 600accepts a swipe input (e.g., a vertical or horizontal input to theparticular query result fields 604) to facilitate browsing the entriesdisplayed in the particular query result fields 604.

The scrubber track 606 facilitates limiting results displayed in thequery result fields 604 by a criterion. In some examples, the criterionis a chronological value (e.g., a date) associated with the entries inthe query result 102. Alternatively, in some examples, the criterion isa relevancy score associated with the entries in the query result 102.The scrubber track 606 may be displayed vertically or horizontally. Theevent points 608 demark events or time periods of significance. In someexamples, the events or the time periods of significance are (a) theevents or time periods that are location maxima of mentions in theentries of the query results, and/or (b) a curate set of eventsassociated with the particular query 104. For example, if the query is“Chicago,” the event points 608 may mark 1803 (the founding of FortDearborn), 1871 (the great Chicago fire), 1893 (the Columbian ExpositionWorld's Fair), 1933 (the Century of Progress World's Fair), and/or 1973(Sears Tower complete), etc. The timeline markers 610 mark points intime (e.g., day, weeks, years, decades, etc.) depending on thechronological scope of the query. For example, the timeline markers 610may mark every fifty years.

The scrubber 612 indicates which of the entries are displayed in thequery result fields 604. The scrubber 612 shows a relative amount ofentries that are being displayed in the query result fields 604. Forexample, if there are 2000 entries and 400 entries are being shown, thescrubber 612 would encompass twenty percent of the scrubber track 606.In some examples, the size of the scrubber 612 is determined by a userpreference for (i) a number of the entries displayed in the query resultfields 604 or (ii) a percentage of entries displayed in the query resultfields 604. The position marker 614 controls the position of thescrubber 612. Changing the location of the position marker 614 along thescrubber track 606 changes at least some of the entries displayed in thequery result fields 604.

FIG. 7 depicts another interface 700 presented by the end-user device106 of FIG. 1 to facilitate presenting and interacting with the queryresults 102. In the illustrated example, the interface 700 presents thequery results 102 in a hierarchical manner. In some examples, theinterface 700 is superimposed on the interface 516-522 from which thequery 104 was generated. The example interface 700 presents a firsthierarchy element 702 that displays a first level of results. In someexamples, the first level of results is the content database providers108. When one of the first levels of results is selected, the interfacepresents a second hierarchy element 704 that displays a second level ofresults. In some examples, the second level of results is the topicsassociated with the query 104. When one of the second levels of resultsis selected, the interface 700 displays a third hierarchy element 706that displays a third level of results. In some examples, the thirdlevel of results includes the entries from the selected content databaseprovider about the selected topic. In some examples, the interface 700may include further hierarchical levels to facilitate browsing the queryresults 102 in finer levels of detail. In the illustrated example, theinterface 700 includes the scrubber track 606, the event points 608, thetimeline markers 610, the scrubber 612, and/or the position marker 614to facilitate navigating though the query results 102 based on thecriterion (e.g., the chronological value, the relevance value, etc.).

FIG. 8 depicts another interface 800 presented by the end-user device106 of FIG. 1 to facilitate presenting and interacting with the queryresults 102. In the illustrated example, the interface includes thefirst hierarchy element 702, the second hierarchy element 704, and thethird hierarchy element 706 arranged in concentric circles. In theillustrated example, the first hierarchy element 702 is associated withthe topic, the second hierarchy element 704 is associated with thecontent database providers 108, and the third hierarchy element 706 isassociated with the entries. Initially, some of the hierarchies 704 and706 are hidden until a selection is made on the associated lower levelhierarchy (e.g., the first hierarchy element 702). In some examples, theinterface 800 includes more hierarchies. In such examples, at thehierarchies closer to the center are more general and the hierarchiesbecome more specific the further from the center they are. An examplehierarchy may be: topic→content providers→entry types→specific entries.In some examples, a user scrolls through the hierarchies 702-706 byturning the hierarchy of interest about a central axis. Although notshown, the interface 800 may include the scrubber track 606, the eventpoints 608, the timeline markers 610, the scrubber 612, and/or theposition marker 614 to facilitate navigating though the query resultsbased on the criterion (e.g., the chronological value, the relevancevalue, etc.).

FIG. 9 depicts the map geometry query interface 518 presented by theend-user device 106 of FIG. 1 to facilitate generating the query 104 andpresenting and interacting with the query results 102. In theillustrated example, the user browses map geometry data and/or staticimages to generate the query 104. Based on the displayed map geometrydata or the static image, the end-user device 106 sends pre-query datato the ARPE 100 to identify objects 900 (e.g., buildings, landmarks,people, etc.) in the map geometry data or the static image. In theillustrated example, the map geometry query interface 518 outlines theidentified objects. When the user interacts with the highlighted object(e.g., the object 900), the map geometry query interface 518 generates aquery 104 about that object.

Initially, in some examples, the map geometry query interface 518 maydisplay a zoomed-out view of an overhead map of the area around thelocation of the end-user device 106. The user may set a preference forhow much the initial view of the overhead map is zoomed-out. The userselects a location on the overhead map to view the map geometry data. Inthe illustrated example, the map geometry query interface 518 includesan inset map 902 that shows (a) the location of the point-of-view on theoverhead map and (b) the orientation of the point-of-view of currentlydisplayed map geometry data in comparison to the overhead map. The mapgeometry query interface 518 may also include other windows to displayinformation about the displayed map geometry data. When the end-userdevice 106 receive results from the ARPE 100, the map geometry queryinterface 518 displays the results in via an interface (e.g., theinterfaces 600, 700, and 800 of FIGS. 6, 7, and 8 above) super imposedon the map geometry query interface 518. In the illustrated example, themap geometry query interface 518 includes results display via thehierarchical interface 700 of FIG. 7 above. In the illustrated example,windows (e.g., the hierarchy elements 702, 704, and 706, the inset map902, a query text box 904, etc.) displayed by the map geometry queryinterface 518 include handles 906 that facilitate hiding and expandingthe windows. In some examples, to hide one of the windows, thecorresponding handle 906 is dragged to an edge of the map geometry queryinterface 518. In some examples, double-tapping the handles 906 causesthe corresponding window (e.g., the inset map 902) to collapse orexpand. In some examples, event points 608 are added to the scrubbertrack 606 to indicate entries (e.g., images, videos, etc.) of the scenedisplayed by the map geometry query interface 518 are available thatrepresent a different point in time.

In some examples, the end-user device 106 includes an accelerometer anda magnetometer to facilitate determining the orientation of the end-userdevice 106. In some such examples, when the map geometry data isdisplayed, changing the orientation of the end-user device 106 changeswhich portion of the map geometry data is displayed. In such a manner,the user may change the map geometry data by moving the end-user device106. In such examples, as the displayed map geometry data changes, theend-user device 106 sends pre-query data to the ARPE 100 to identify theobjects 900 in the displayed map geometry data or the static image. Themap geometry query interface 518 continually identifies (via thepre-query responses) and outlines the objects 900.

FIG. 10 depicts the map query interface 520 presented by the end-userdevice 106 of FIG. 1 to facilitate generating the query 104 andpresenting and interacting with the query results 102. In theillustrated example, the map query interface 520 displays a map 1000.The map 1000 may be selected via one of the result interface (e.g.,interfaces 600, 700, and 800 of FIGS. 6, 7, and 8 , above). The mapquery interface 520 includes the scrubber track 606, the event points608, the timeline markers 610, the scrubber 612, and/or the positionmarker 614 to facilitate navigating though related maps (e.g., maps thatcover the same or substantially similar geographical areas, etc.) withdifferent chronological values. The event points 608 are indicative ofmaps with different chronological values. The illustrated exampleincludes a trackhead menu 1004 to display the map(s) associated with theevent point 608 at which the position marker 614 is located. In someexamples, the trackhead menu 1004 presents a string list to facilitateselecting different maps associated with the event point 608.

The map query interface 520 includes highlighted regions 1002 toindicate suggested queries 104. For example, a map that depicts Chicagoin 1893 may include a highlighted region 1002 around the area of the mapof the World's Columbian Exposition. When one of the highlighted regions1002 is selected, the map query interface 520 generates a query 104associated with the topic of the corresponding highlighted region 1002.The query results 102 may be superimposed on the map query interface 520using one of the result interfaces 600, 700, and 800. In some examples,the map query interface 520 includes a view distance control panel 1006that facilitates increasing and decreasing the magnification or zoom ofthe map 1000. In some such examples, if the user set a view distance toshow regions beyond the borders of the map 1000, the map query interface520 display (a) other maps that have a similar chronological valueand/or (b) current maps (such as satellite imagery maps, roadmaps,etc.).

FIG. 11 depicts an augmented reality query interface 522 presented bythe end-user device 106 of FIG. 1 to facilitate generating the query 104and presenting and interacting with the query results 102. In theillustrated example, the augmented reality query interface 522 displaysa live image 1102 captured by one of the cameras (e.g., the cameras 2614and 2616 of FIG. 26 below). Additionally or alternatively, in someexamples, the augmented reality query interface 522 displays images froma media application (e.g., Netflix, Hulu, local broadcasts via a smartTV, etc.) as if the images were being captured by one of the cameras. Insome examples, when the augmented reality query interface 522 isdescribed below as analyzing the live images captured by one of thecameras, the augmented reality query interface 522 may instead beanalyzing images from the media application. From time-to-time (e.g.,periodically, aperiodically, etc.), the augmented reality queryinterface 522 sends pre-query data to the ARPE 100 to identify objects1104 within the live image 1102. The information in the pre-query datadepends on the sensors of the end-user device 106. When an object 1104is recognized, the augmented reality query interface 522 outlines and/orhighlights the object 1104. In some examples, when an object 1104 isrecognized, the ARPE 100 performs abbreviated queries with the objects1104 as topics. In such an example, the augmented reality queryinterface 522 displays abbreviated query panels 1106 with basicinformation (e.g., identity of the object, dates associated with theobject, number of entries pertaining to the object, factoids about theobject, available subscriptions to content database providers 108 withinformation about the object, etc.). When the user interacts with theoutlined and/or highlighted area or the abbreviated query panels 1106,the augmented reality query interface 522 generates and sends the query104 with the identity of the object as the topic to the ARPE 100.

When the augmented reality query interface 522 receives the queryresults 102, the augmented reality query interface 522 may super imposethe query results based on the results interface (e.g., the interfaces600, 700, and 800 of FIGS. 6, 7, and 8 above). Additionally, in someexamples, in response to an entry from the query results 102 beingselected, the augmented reality query interface 522 super imposes theselected entry onto the live image 1102. For example, if the selectedentry is an image associated with coordinates and an orientation, theaugmented reality query interface 522 may transform (e.g. change size,display angle, and/or perspective, etc.) the image to display in theplace of the corresponding object 1104. In some examples, the augmentedreality query interface 522 emphasizes local entries (e.g., related totopics that are geographically proximate to the location of the end-userdevice 106).

Additionally, in some examples, the augmented reality query interface522 facilities generating entries to be associated with the outlinedand/or highlighted the objects 1104. In such an example, the user, viaan input device (e.g., the input devices 2608 of FIG. 26 below) enters amessage (e.g., a length limited message) to be pinned to (e.g.,associated in the ARP database 112 with) the selected outlined and/orhighlighted the object 1104 and associated with a date. Additionally,the augmented reality query interface 522 may receive the message aspart of the pre-query data and present (e.g., via one of the interfaces600, 700, and 800) the pinned messages when the corresponding object1104 is outlined and/or highlighted.

FIG. 12 is a block diagram of the database sorter 110 of FIG. 1 thatgenerates the entry records 1200 (e.g., based on the entry record datastructures 200 of FIG. 2 ). In the illustrated example, the databasesorter 110 includes an example content scanner 1202, an example imageidentifier 1204, an example content tagger 1206, and an example recordcreator 1208.

The example content scanner 1202 receives entries 1210 from the contentdatabase providers 108. The content scanner 1202 forwards ones of theentries 1210 that are images to the example image identifier 1204. Thecontent scanner 1202 analyzes the entries 1210 to identify topics ofinterest within the entries 1210. The content scanner 1202 analyzes thebody of the entry 1210, the title of the entry 1210, and metadata (e.g.,tags, author, dates, etc.), etc. In the illustrated example, thedatabase sorter 110 maintains an example topic database 1212 thatincludes topics on interest to be identified by the content scanner1202. In some examples, the topic database 1212 includes abbreviationsand common misspellings of the topics of interest. In some examples, thecontent scanner 1202 analyzes the entries to determine potential topicsthat are not in the topic database 1212. The potential topics are wordsin sentences that may be the subject or object of the sentence. In somesuch example, the potential topics may be flagged to determine whetherthe topic should be added to the topic database 1212. For example, ifthe entry 1210 states “The dogtor prescribed medicine to my corgi,” thecontent scanner 1202 may (a) identify {medicine, corgi} as the topics ofthe entry 1210, and (b) {dogtor} as a potential topic. In some examples,when the entry 1210 does not have a text body (e.g., is a video, anaugmented reality experience, etc.), the content scanner 1202 identifiestopics in the metadata of the entry 1210 which may include topicssuggested by the corresponding content database provider 108.Additionally, the content scanner 1202 analyzes the entries 1210 toidentify dates associated with the entries. For example, an entry 1210written on Apr. 4, 2008 that discusses the battle at Gettysburg may beassociated with that dates of {Jul 1, 1863, Jul. 1, 1863, Jul. 1, 1863,Apr. 4, 2008}.

The image identifier 1204 identifies topics within the entries 1210 thatinclude images. The image identifier 1204 analyzes the metadata (e.g.,HTML data, the EXIF data, the XMP data, etc.) associated with the imageto determine tags of topics associated with the image. For example, theXMP data associated with the image may include keywords, key phrases,location markers, coordinates, or classification codes that describe thetopic(s) of the image. In some example, the image identifier 1204 usesmachine learning techniques (such as neural networks, etc.) to performsimage recognition to identify topics in the image. Additionally, in someexamples, the image identifier 1204 performs facial recognition toidentify people in the image. In some examples, the image identifier1204 performs a reverse image lookup to (i) identify other entries inwhich the image is used and/or (ii) identify other copies of the imagethat may contain different metadata. In some such examples, the imageidentifier 1204 identifies topics within the image based on the topicsin the other associated entries and/or images. In some such examples,the topics are weighted based on the source (e.g., the content databaseprovider 108) of the other associated entries and/or images. Forexample, the image identifier 1204 may distinguish between professionalsources (e.g., professional image providers, such as Getty® Images,newspapers, etc.) and crowdsourced sources (e.g., Wikipedia, AmazonTurk, etc.).

The example content tagger 1206 associates the topics in the entries1210 identified by the content scanner 1202 and/or the image identifier1204 with topic identifiers for the topic ID field 206. In theillustrated example, the content tagger 1206 uses a synonym database1214. The synonym database 1214 (a) associates topics that are synonyms,and (b) associates the topic identifiers with topics identified by thecontent scanner 1202 and/or the image identifier 1204. For example, thetopics of {Abraham Lincoln, Honest Abe, Uncle Abe, The GreatEmancipator, the Illinois Rail Splitter} may be associated with thetopic identifier {713589}. In some examples, the content tagger 1206includes the identified topic identifier(s) in the topic ID field 206for a particular entry record if a number of times the topic or one ofits synonyms is identified in the entry 1210 satisfies (e.g., is greaterthan or equal to) a threshold. In some such examples, the threshold isbased on a number of topics identified in the entry 1210. For example,if thirty-five topics are identified in the entry 1210, the thresholdmay be five. Additionally, in some examples, the content tagger 1206calculates relevance scores for the topics to be associated with theentry 1210. The relevance score measures the importance of the topic tothe entry 1210. For example, the relevance score for text-based entries1210 may be based on (i) the frequency of the topic in entry 1210, (ii)the length of the entry 1210, and (iii) the number of entries that 1210that contain the topic.

The example record creator 1208 creates the entry records 1200 based onthe entry record data structures 200. The record creator 1208 generatesa unique record identifier to be entered into the record ID field 202.The record creator 1208 enters an identifier associated with thecorresponding content database provider 108 in the CDP ID field 204.Additionally, the record creator 1208 enters the topic identifiersidentified by the content tagger 1206 into the topic ID field 206. Therecord creator 1208 enters metadata associated with the entry 1210(e.g., the title, coordinates, the author, the file size, etc.) into thetopic metadata field 208. In some examples, the record creator 1208maintains a metadata database 1216 that includes information (e.g.,coordinates associated with a topic, dates associated with a topic,etc.) about the topics that may not be included in the entry 1210. Therecord creator 1208 enters dates associated with the topic identified bythe content scanner 1202 and/or the image identifier 1204 or included inthe metadata database 1216 into the topic date field 210. The recordcreator 1208 enters the URI associated with the entry 1210 into the URIfield 212. The record creator 1208 then stores the entry record 1200into the ARP database 112.

FIG. 13 is a block diagram of the query responder 120 of the augmentedreality platform entity 100 of FIG. 1 . In the illustrated example, thequery responder 120 includes an example query response generator 1300,an example query organizer 1302, and an example reply generator 1304.

The example query response generator 1300 receives the query 104 fromthe end-user device 106 via the session handler 118. The query responsegenerator 1300 analyzes the query 104 to determine the topic(s)contained in the query 104. In some examples, because some topics maynot be directly contained in the query 104 (e.g., are inferential), thequery response generator 1300 determines some topics based on inferencesto other topics contained in the query 104. For example, if the querydata field 306 of the query include the text string “person who designedthe rookery building,” the query response generator 1300 may retrievepeople-related topics from the topic database that are related to thetopic of the “Rookery Building.” Based on the topics, the query responsegenerator 1300 determines which topics in the topic database 1212 arerelated (e.g., have affinity with) the topics of the query 104.

The example query organizer 1302 receives and/or otherwise retrieves thetopics identified by the query response generator 1300. Based on thetopics, the query organizer 1302 retrieves the entry records 1200related to the topics from the ARP database 112. In some examples, theentry records 1200 are filtered based on (i) the relevance of the entryrecords 1200 to the identified topics, and (ii) relationship between theidentified topics and the query 104.

The example reply generator 1304 generates the query results 102 basedon the entry records 1200 received and filtered by the query organizer1302. Based on user preferences stored in the user database 116, thereply generator 1304 filters and orders the entry records 1200 based onthe preferences. For example, a first user may prefer entries bepresented by topic and then database content provider 108 or vice versa.In some examples, the reply generator 1304 assigns the records intochronological categories that are associated with periods of time. Insome such examples, the chronological categories are based on a numberof dates associated with the entry records 1200 in the query results 102and clustering of the dates. Additionally, in some examples, the replygenerator 1304 organizes the entry records 1200 into the chronologicalcategories. In such examples, the reply generator 1304 organizes theentry records 1200 in each of the chronological categories intohierarchies (e.g., the hierarchies 702, 704, and 706 of FIG. 7 above).The reply generator 1304 formats (e.g., in HTML, in XML, etc.) theorganized entry records 1200 to generate the query results 102.

FIG. 14 is a block diagram of a query generator 1400 of the end-userdevice 106 of FIG. 1 . The query generator 1400 generates queries 104based on input from the end-user device 106 (e.g. via the input devices2608 and/or the cameras 2614 and 2616 of FIG. 26 below) and presents thequery results 102 via a display (e.g., the display 2612 of FIG. 26below). In some examples, the query generator 1400 is included in anapplication that is downloaded onto the end-user device 106 from anapplication store (e.g., the Google Play Store, the App Store, etc.).Alternatively, in some examples, the query generator 1400 is implementedin hardware and software. In the illustrated example, the querygenerator 1400 includes an example query selector 1402, an example queryformatter 1404, and an example result presenter 1406.

The query selector 1402 presents an interface (e.g., one of theinterfaces 502-512) to the user on a display (e.g., the display 2612 ofFIG. 26 below). The query selector 1402 receives input (e.g., from theinput devices 2608 and/or the cameras 2614 and 2616 of FIG. 26 below) tofacilitate a user navigating between the interfaces 502-512 and the userselecting which one of the content navigation interfaces 512 to generatea query 104.

The query formatter 1404 receives input to determine the query of theuser based on the content navigation interfaces 512 selected via thequery selector 1402. For some interfaces (e.g., the map geometry queryinterface 518, the map query interface 520, the augmented reality queryinterface 522, etc.), the query formatter 1404 generates pre-query data1408 to be sent to the ARPE 100 to provide context to the selectedinterface. The pre-query data 1408 provides context to the ARPE 100about the location and environment of the end-user device 106. Forexample, the pre-query data 1408 may include GPS coordinates, directionheading, airspeed, and altitude to provide context for objects ofinterest around when the end-user device 106 is used on a plane. Thetype of information included in the pre-query data 1408 may be based onthe interface 512 being used. For example, for the augmented realityquery interface 522, the query formatter 1404 may generate pre-querydata 1408 that includes image(s) captured by the camera(s) of theend-user device 106. Additionally, the query formatter 1404 based oninput from a user and, in some example, a pre-query response 1410, thequery formatter 1404 generates the query 104.

The result presenter 1406 receives the query results 102 and, in someexamples, the pre-query response 1410. The result presenter 1406superimposes the query results 102 onto the interface selected via thequery selector 1402. In some examples, in response to receiving thepre-query response 1410, the result presenter 1406 superimposes outlinesand/or highlights of topics of interest contained in the pre-queryresponse 1410 on the interface selected via the query selector 1402. Forexample, if the pre-query response 1410 includes data for a particulartombstone from an image in the pre-query data 1408, the result presenter1406 superimposes an outline and/or a highlight the tombstone in theinterface being displayed to the user.

FIG. 15 is a flowchart of a method to index entries 1210 from contentdatabase providers 108, which may be implemented by the processorplatform of FIG. 25 . Initially, at block 1502, the content scanner 1202receives an entry 1210 from one of the content database providers 108.At block 1504, the content scanner 1202 determines whether the entry1210 received at block 1502 is an image. If the entry 1210 is an image,the method continues at block 1506. Otherwise, if the entry 1210 is notan image, the method continues at block 1508.

At block 1506, the image identifier 1204 analyzes the image to determinethe topics in the image. The image identifier 1204 evaluates themetadata of the image for tags that identify the topics depicted by theimage. In some examples, the image identifier 1204 performs imagerecognition on the image to identify the topics depicted by the image.At block 1508, the content scanner 1202 analyzes the text and metadataassociated with the entry 1210 to determine the topics of interest inthe entry 1210. At block 1510, the content tagger 1206 tags theidentified topics identified by the content scanner 1202 and/or theimage identifier 1204. In some examples, based on the synonym database,the content tagger 1206 consolidates the topics identified by thecontent scanner 1202 and/or the image identifier 1204. Additionally, atblock 1512, the content tagger 1206 identifies dates associated with theentry and/or calculates a relevance score for each of the identifiedtopics. At block 1514, the record creator 1208 creates an entry record1200 for the entry 1210 including the content database provider 108, theidentified topics, the identified dates, the URI, and identifyingmetadata. At block 1516, the record creator 1208 stores the entry record1200 in the ARP database 112.

The flowchart of FIG. 15 is representative of machine readableinstructions stored in memory (such as the memory 2504 of FIG. 25 below)that comprise one or more programs that, when executed by a processor(such as the processor 2502 of FIG. 25 below), cause the ARPE 100 toimplement the example query responder 120 of FIGS. 1 and 13 . Further,although the example program(s) is/are described with reference to theflowchart illustrated in FIG. 15 , many other methods of implementingthe example query responder 120 may alternatively be used. For example,the order of execution of the blocks may be changed, and/or some of theblocks described may be changed, eliminated, or combined.

FIG. 16 is a flowchart of a method to generate the queries 104 andpresent the query results 102, which may be implemented by theelectronic components of FIG. 26 . Initially, at block 1602, the queryselector 1402 displays a text query bar and query options via the homeinterface 506. At block 1604, the query selector waits until a type ofquery has been selected. At block 1606, the query selector 1402 displaysthe selected interface (e.g., one of the interfaces 516-522). At block1608, the query selector 1402 waits until the user enters (e.g., enterstext into a text box, selects an identified object on map geometry data,selects a point of interest on a map, selects an object identified in alive image, etc.). At block 1610, the query formatter 1404 generates thequery 104 and sends the query 104 to the ARPE 100. At block 1612, theresult presenter 1406 receives query results 102 from the ARPE 100. Atblock 1614, the result presenter 1406 displays the query results 102.Example methods to display the query results are disclosed in FIGS.18-22 below.

FIG. 17 is a flowchart of a method to present the query results 102,which may be implemented by the electronic components of FIG. 26 .Initially, at block 1702, the result presenter 1406 displays thescrubber track 606 with event points 608 and/or timeline markers 610.The location of the event points 608 along the scrubber track 606 arebased on dates associated with the query results 102. For example, theevent points 608 may represent a number (e.g., five, ten, twenty, etc.)of the most reoccurring dates within the query results 102. For example,if the query results 102 are based on the topic “the battle ofGettysburg,” a first event point 608 may represent Jul. 1, 1863, asecond event point 608 may represent Jul. 2, 1863, a third event point608 may represent Jul. 3, 1863, and a fourth event point 608 mayrepresent Oct. 19, 1863. In some examples, the result presenter 1406displays the timeline markers 610 based on the earliest date and thelatest date associated with the query results 102. In some such example,the timeline markers 610 are displayed linearly. Alternatively, in someexamples, the timeline markers 610 are displayed logarithmically.

At block 1704, the result presenter 1406 displays the scrubber 612 toindicate scope of currently displayed results. For example, if twentypercent of the results from the query results 102 are displayed, thenthe scrubber 612 may occupy twenty percent of the length of the scrubbertrack 606. At block 1706, the result presenter 1406 displays theposition marker 614 used to modify the location of the scrubber 612 onthe scrubber track 606 which changes of currently displayed results.

At block 1708, the result presenter 1406 selects the next contentdatabase provider 108 (as identified by the CDP ID filed 204) identifiedin the query results 102. At block 1710, the result presenter 1406 sizesthe query result field 604 corresponding to the selected contentdatabase provider 108 based on the number of entries associated with thecontent database provider 108 within the scope of currently displayedresults (as selected via the scrubber 612). At block 1712, the resultpresenter 1406 displays the entries within the corresponding queryresult field 604 associated with (a) the selected content databaseprovider 108 and (b) the current scope as indicated by the scrubber 612.At block 1714, the result presenter 1406 determines whether there isanother content database provider 108 to display within the queryresults 102. If there is another content database provider 108 todisplay, the method returns to block 1708. Otherwise, if there is notanother content database provider 108 to display, the method continuesto block 1716.

At block 1716, the result presenter 1406 determines whether it receivedinput to change which portion of the results are displayed in one of thecategory boxes. For example, the end-user device 106 with a touch screenmay receive a vertical swiping motion within an area of one of the queryresult fields 604. If input is received, the method continues to block1718. Otherwise, if the input is not received, the method continues toblock 1720. At block 1718, the result presenter 1406 changes which ofthe entries are displayed in the query result field 604 associated withthe input. At block 1720, the result presenter 1406 determines whetherit received input to change the position of the scrubber 612. If theinput is received, the method returns to block 1708. Otherwise, if theinput is not receivers, the method returns to block 1716.

FIG. 18 is a flowchart of a method to present the query results 102,which may be implemented by the electronic components of FIG. 26 .Initially, at block 1802, the result presenter 1406 displays thescrubber track 606 with event points 608 and/or timeline markers 610.The location of the event points 608 along the scrubber track 606 arebased on dates associated with the query results 102. In some examples,the result presenter 1406 displays the timeline markers 610 based on theearliest date and the latest date associated with the query results 102.In some such example, the timeline markers 610 are displayed linearly.Alternatively, in some examples, the timeline markers 610 are displayedlogarithmically.

At block 1804, the result presenter 1406 displays the scrubber 612 toindicate scope of currently displayed results. For example, if twentypercent of the results from the query results 102 are displayed, thenthe scrubber 612 may occupy twenty percent of the length of the scrubbertrack 606. At block 1806, the result presenter 1406 displays theposition marker 614 used to modify the location of the scrubber 612 onthe scrubber track 606 which changes of currently displayed results.

At block 1808, the result presenter 1406 displays, in the firsthierarchy element 702, a first group associating content databaseproviders 108 with numbers of entries that (a) are associated with thecontent database provider 108 and (b) are within the scope indicated bythe scrubber 612. At block 1810, the result presenter 1406 determineswhether it has received a selection of one of the content databaseprovider 108. If a selection has been received, the method continues atblock 1812. Otherwise, if a selection has not been received, the methodcontinues at block 1818. At block 1812, the result presenter 1406displays, in the second hierarchy element 704, a second groupassociating topics in the query results 102 corresponding to theselected content database provider 108 with a number of entries that areassociated with topics with the scope indicated by the scrubber 612. Atblock 1814, the result presenter 1406 determines whether it has receiveda selection of one of the topics in the second hierarchy element 704. Ifa selection has been received, the method continues at block 1816.Otherwise, if a selection has not been received, the method continues atblock 1818. At block 1816, the result presenter 1406, in the thirdhierarchy element 706, displays a third group with the entriesassociated with the selected topic separated by type (e.g., articles,maps, images, videos, augmented reality experiences, map geometry data,etc.).

At block 1818, the result presenter 1406 determines whether the inputhas been received to change the position of the scrubber 612. If inputhas been received, the method continues to block 1820. Otherwise, ifinput has not been received, the method returns to block 1810. At block1820, the result presenter 1406 changes the first group displaying thecontent database providers 108 in the first hierarchy element 702, thesecond group displaying the topics in the second hierarchy element 704,and/or the third group displaying the entries in the third hierarchyfield 706 to display results associated with the scope indicated by thescrubber 612.

FIG. 19 is a flowchart of a method to present the query results 102,which may be implemented by the electronic components of FIG. 26 .Initially, at block 1902, the result presenter 1406 displays thescrubber track 606 with event points 608 and/or timeline markers 610.The location of the event points 608 along the scrubber track 606 arebased on dates associated with the query results 102. In some examples,the result presenter 1406 displays the timeline markers 610 based on theearliest date and the latest date associated with the query results 102.In some such example, the timeline markers 610 are displayed linearly.Alternatively, in some examples, the timeline markers 610 are displayedlogarithmically.

At block 1904, the result presenter 1406 displays the scrubber 612 toindicate scope of currently displayed results. For example, if twentypercent of the results from the query results 102 are displayed, thenthe scrubber 612 may occupy twenty percent of the length of the scrubbertrack 606. At block 1906, the result presenter 1406 displays theposition marker 614 used to modify the location of the scrubber 612 onthe scrubber track 606 which changes of currently displayed results.

At block 1908, the result presenter 1406 displays, the first hierarchyelement 702 shaped as a circle or polygon, one of the topics related tothe query 104. In some examples, the result presenter 1406 displaysmultiple first hierarchy elements 702, which may be moveable, resizableand/or hidable, etc., for the different ones of the topics related tothe query 104. For example, if the query 104 is “The Rookery Building,”a first one of the first hierarchy elements 702 may be associated with“The Rookery Building” and a second one of the first hierarchy elements702 may be associated with “Daniel Burnham.”

At block 1910, the result presenter 1406 displays, on the secondhierarchy element 704 shaped as a first ring around the first hierarchyelement 702, the content database providers 108 associated with entriescorresponding to the topic that are within the scope indicated by thescrubber 612. At block 1912, the result presenter 1406 determineswhether a selection of one content database providers 108 has beenreceived. If a selection has been received, the method continues toblock 1914. Otherwise, if a selection has not been received, the methodcontinues to block 1916.

At block 1914, the result presenter 1406 displays, on the thirdhierarchy element 706 shaped as a first ring around the second hierarchyelement 704, display entries associated with the selected contentdatabase provider 108 that are within the scope indicated by thescrubber 612. At block 1916, the result presenter 1406 determineswhether the input has been received to change the position of thescrubber 612. If input has been received, the method continues to block1918. Otherwise, if input has not been received, the method returns toblock 1912. At block 1918, the result presenter 1406 changes the contentdatabase providers 108 in the second hierarchy element 704 and/or theentries in the third hierarchy element 706 to display entries associatedwith the scope indicated by the scrubber 612.

FIG. 20 is a flowchart of a method to present the query results 102 inconjunction with map geometry data, which may be implemented by theelectronic components of FIG. 26 . Initially, at block 2002, the queryselector 1402 displays the map geometry. The map geometry includesimages that may be planned to illustrate a geographical location as ifthe user of the end-user device 106 was at that location. At block 2004,the query selector 1402 displays an inset map 902 indicating theorientation of the displayed map geometry. At block 2006, the queryformatter 1404 sends pre-query data 1408 to the ARPE 100. In someexamples, the pre-query data 1408 includes an image being displayed,coordinates from which the map geometry data was captured, and the angleat which the map geometry data is being viewed. At block 2008, the queryselector 1402 outlines places or things in the displayed at block 2002recognized in the map geometry data. At block 2010, the query selector1402 determines whether it has received a selection of one of the placesor things outlines at block 2008. If a selection has been received, themethod continues at block 2012. Otherwise, if a selection has not beenreceived, the method continues at block 2016.

At block 2012, the query formatter 1404 sends the query 104 to the ARPE100 with the selected place or thing in the query data field 306. Atblock 2014, the result presenter 1406 displays the query results 102received from the ARPE 100. Examples methods of displaying the queryresults 102 are disclosed in FIGS. 17, 18, and 19 above. At block 2016,the query selector 1402 determines whether it has received input tochange the displayed map geometry data. If the input to change thedisplayed map geometry data has been received, the method returns toblock 2002. Otherwise, if the input to change the displayed map geometrydata has not been received, the method returns to block 2010.

FIG. 21 a flowchart of a method to present the query results 102 inconjunction with map data, which may be implemented by the electroniccomponents of FIG. 26 . Initially, at block 2102, the query selector1402 displays a map retrieved via a query 104. For example, one of theentries include in query results 102 may be a map 1000. At block 2104,the query selector 1402 displays event points 608 along the scrubbertrack 606 indicating other related maps. For example, if the displayedmap 1000 is a map of Chicago from 1893, the query selector 1402 maydisplay event points 608 along the scrubber track 606 indicating maps ofChicago associated with other dates. At block 2106, the query selector1402 determines whether the displayed map 1000 is associated withlocations of interest. In some examples, the information (e.g.,boundaries on the map, related topics, etc.) about the locations ofinterest is included in the map data. If there are locations of interestassociated with the displayed map 1000, the method continues at block2108. Otherwise, if there are not locations of interest associated withthe displayed map 1000, the method continues to block 2116.

At block 2108, the query selector 1402 highlights locations of intereston the map 1000 (e.g., the highlighted regions 1002 of FIG. 10 ). Atblock 2110, the query selector 1402 determines whether one of thelocations of interest has been selected. If one of the locations ofinterest has been selected, the method continues at block 2112.Otherwise, if one of the locations of interest has not been selected,the method continues at block 2116. At block 2112, the query formatter1404 sends a query 104 to the ARPE 100 with location of interest in thequery data field 306. At block 2114, the result presenter 1406 displaysthe query results 102 received from the ARPE 100. Examples methods ofdisplaying the query results 102 are disclosed in FIGS. 17, 18, and 19above. At block 2116, the query selector 1402 determines whether theposition of the position marker 614 changed. If the position of theposition marker 614 changed, the method continues to block 2118.Otherwise, if the position of the position marker 614 did not change,the method returns to block 2110. At block 2118, the query selector 1402displays a map 1000 associated with position of the position marker 614.

FIG. 22 a flowchart of a method to present the query results 102 inconjunction with live image data from a camera (e.g., the camera 2614and 2616 of FIG. 26 below), which may be implemented by the electroniccomponents of FIG. 26 . Initially, at block 2202, the query selector1402 displays the image captured by the camera. At block 2204, the queryformatter 1404 sends pre-query data 1408 to the ARPE 100. In someexamples, the pre-query data includes the image captured by the camera,the coordinates of the end-user device 106, the orientation of thecamera, and/or the field of view of the camera, etc. At block 2206, thequery selector 1402 outlines places or things recognized in the imagebased on a pre-query response 1410 received from the ARPE 100. At block2208, the query selector 1402 determines whether one of the outlinedplaces or things has been selected. If one of the outlined places orthings has been selected, the method continues at block 2210. Otherwise,if one of the outlined places or things has not been selected, themethod continues at block 2214.

At block 2210, the query formatter 1404 sends a query 104 to the ARPE100 with the selected place or thing in the query data field 306. Atblock 2212, the result presenter 1406 displays the query results 102received from the ARPE 100. Examples methods of displaying the queryresults 102 are disclosed in FIGS. 17, 18, and 19 above. At block 2214,the query selector 1402 determines whether (a) the image captured by thecamera has changed and/or (b) the pose (e.g., coordinates andorientation, etc.) of the camera has changed. If (a) the image capturedby the camera has changed and/or (b) the pose of the camera has changed,the method returns to block 2204. Otherwise, if (a) the image capturedby the camera has not changed and/or (b) the pose of the camera has notchanged, the method returns to block 2208.

The flowcharts of FIGS. 16-22, and 24 are representative of machinereadable instructions stored in memory (such as the memory 2604 of FIG.26 ) that comprise one or more programs that, when executed by aprocessor (such as the processor 2602 of FIG. 26 ), cause the end-userdevice 106 to implement the example query generator 1400 of FIG. 14 .Further, although the example program(s) is/are described with referenceto the flowchart illustrated in FIGS. 16-22, and 24 , many other methodsof implementing the example query generator 1400 may alternatively beused. For example, the order of execution of the blocks may be changed,and/or some of the blocks described may be changed, eliminated, orcombined.

FIG. 23 illustrates the augmented reality interface 522 of FIG. 11providing location specific content 2302 superimposed on live imagescaptured by the end-user device 106 of FIG. 1 . The end-user device 106provides its location to the ARPE 100 (e.g., via the pre-query data1408). The ARPE 100 determines whether one of the content databaseproviders 108 associated with a venue (e.g., a stadium, a theater, amuseum, etc.) corresponding to the location of the end-user device 106is providing location specific content 2302. The location specificcontent 2302 is content that is being generated in near real-time (e.g.,there may be a delay between the creation of the content for processingand/or indexing, etc.) for users located at the venue. For example, thelocation specific content may be closed captioning of an announcer or atour guide, etc. In some examples, the ARPE 100 provides the locationspecific content 2302 via the pre-query response 1410. In some suchexamples, as long as enabled, the ARP 100 continuously pushes thelocation specific content 2302 to the end-user device 106 via thepre-query response 1410. In such a manner, the venue may provideservices to cater to the needs of users while facilitating the usersviewing the event or exhibit uninterrupted.

If enabled, the location specific content 2302 is displayed by theaugmented reality interface 522 by superimposing the location specificcontent 2302 on the image being captured by the camera(s) (e.g., thecameras 2614 and 2616 of FIG. 26 below) of the end-user device 106. Insome examples, the location specific content 2302 is displayed on aspecific highlighted and/or outlined object 1104. For example, thelocation specific content 2302 may be displayed so that it is visible ona scoreboard or jumbotron of a stadium. In the illustrated example, theARPE 100 (e.g., via the database sorter 110) provides abbreviated querypanels 1106 for topics detected within the location specific content2302. When the user interacts with the abbreviated query panels 1106,the augmented reality query interface 522 generates and sends the query104 regarding the corresponding topic to the ARPE 100.

In some examples, the ARPE 100 provides the location specific content2302 to the end-user device 106 when the end-user device 106 is notlocated at the venue, but is viewing content (e.g., via a mediaapplication) that is generated at the venue. For example, a user may beviewing a baseball game via a media application on a smart TV. In suchexamples, the augmented reality interface 522 uses the media applicationas the input (e.g., instead of the images being captured by thecamera(s) etc.) and superimposes the location specific content 2302 onthe media application.

FIG. 24 is a flowchart of a method to detect and provide locationspecific content 2302 superimposed on live images captured by theend-user device 106 of FIG. 1 , which may be implemented by theelectronic components of FIG. 26 . Initially, at block 2402, the queryformatter 1404 determines the location of the end-user device 106 (e.g.,via the GPS receiver 2620 of FIG. 26 below). In some examples, the queryformatter 1404 sends pre-query data 1408. At block 2404, the queryformatter 1404 determines whether the location specific content 2302 isavailable. In some examples, the query formatter 1404 receives anindication of whether the location specific content 2302 is available inthe pre-query response 1410. If the location specific content 2302 isavailable, the method continues to block 2406. Otherwise, is thelocation specific content 2302 is not available, the method returns toblock 2402.

At block 2406, the result presenter 1406 presents the augmented realityinterface 522 of FIG. 5 on the end-user device 106. At block 2408, thequery formatter 1404 retrieves or otherwise receives the locationspecific content 2302. In some examples, the location specific content2302 is included in the pre-query responses 1410. At block 2410, theresult presenter 1406 superimposes the location specific content 2302onto the image displayed by the end-user device 106. At block 2412, theresult presenter 1406 determines whether a topic is identified in thelocation specific content 2302. If a topic is identified, the methodcontinues at block 2414. Otherwise, if a topic is not identified, themethod continues at block 2416.

At block 2414, the result presenter 1406 displays the abbreviated querypanels 1106 in associated with the topic identified in the locationspecific content 2302. The information for the abbreviated query panels1106 is included in the pre-query responses 1410. At block 2416, thequery formatter 1404 determines whether there is more location specificcontent 2302. If there is more location specific content 2302, themethod returns to block 2408. Otherwise, if there is not more locationspecific content 2302, the method returns to block 2402.

FIG. 25 is a block diagram of an example processor platform 2500 capableof executing the instructions to implement the method of FIG. 15 . Theprocessor platform 2500 can be, for example, a server, a personalcomputer, a workstation, one or more virtual machines and/or containers,or any other type of computing device. In the illustrated example, theprocessor platform 2500 includes a processor 2502, memory 2504, storage2506, input devices 2508, output devices 2510, and network devices 2512.

The processor platform 2500 of the illustrated example includes theprocessor 2502. The processor or controller 2502 may be any suitableprocessing device or set of processing devices such as, but not limitedto: a microprocessor, a controller-based platform with multipleprocessing cores, a suitable integrated circuit, one or more fieldprogrammable gate arrays (FPGAs), and/or one or moreapplication-specific integrated circuits (ASICs). In the illustratedexample, the processor 2502 is structured to include the exampledatabase sorter 110, the example user handler 114, the example sessionhandler 118, and the example query responder 120.

The memory 2504 may be volatile memory (e.g., RAM, which can includenon-volatile RAM, magnetic RAM, ferroelectric RAM, and any othersuitable forms); non-volatile memory (e.g., disk memory, FLASH memory,EPROMs, EEPROMs, memristor-based non-volatile solid-state memory, etc.),unalterable memory (e.g., EPROMs), and/or read-only memory. In someexamples, the memory 2504 includes multiple kinds of memory,particularly volatile memory and non-volatile memory. The storage 2506includes high-capacity storage devices, such as hard drives, tapedrives, and/or solid state drives, etc. In the illustrated example, thestorage 2506 includes the ARP database 112, the user database 116, thetopic database 1212, the synonym database 1214, and/or the metadatadatabase 1216. When used by the processor 2502, portions of thedatabases 112, 116, 1212, 1214, and 1216 may be move into the memory2504, processed by the processor 2502, and then re-stored in the storage2506.

The memory 2504 and storage are computer readable media on which one ormore sets of instructions, such as the software for operating themethods of the present disclosure can be embedded. The instructions mayembody one or more of the methods or logic as described herein. In aparticular embodiment, the instructions may reside completely, or atleast partially, within any one or more of the memory 2504, the storage2506, and/or within the processor 2502 during execution of theinstructions.

The terms “non-transitory computer-readable medium” and“computer-readable medium” should be understood to include a singlemedium or multiple media, such as a centralized or distributed database,and/or associated caches and servers that store one or more sets ofinstructions. The terms “non-transitory computer-readable medium” and“computer-readable medium” also include any tangible medium that iscapable of storing, encoding or carrying a set of instructions forexecution by a processor or that cause a system to perform any one ormore of the methods or operations disclosed herein. As used herein, theterm “computer readable medium” is expressly defined to include any typeof computer readable storage device and/or storage disk and to excludepropagating signals, to exclude transitory signals, and to excludetransmission media. As used herein, the term “non-transitory” refers tostoring information on the computer readable medium for any duration(e.g., permanently, for long durations (e.g., minutes, days, etc.), fortemporarily buffering and/or for caching, etc.).

The example input devices 2508 include any device that facilitates auser interacting with the processor platform 2500 to, for example, enterdata and commands. The input devices 2508 include, for example, amicrophone, a camera, a keyboard, a button, a mouse, a touch screen, acontroller, a gesture recognition device, and/or a voice recognitionsystem.

The example output devices 2510 include any device to facilitatecommunication of data and/or status to a user. Example output devices2510 include instrument cluster outputs (e.g., dials, lighting devices),actuators, a heads-up display, a display (e.g., a liquid crystal display(“LCD”), an organic light emitting diode (“OLED”) display, a flat paneldisplay, a solid state display, etc.), and/or speakers.

The example network devices 2512 include any suitable communicationdevice that facilitates communication with the end-user devices 106 overan external network. The external network(s) may be a public network,such as the Internet; a private network, such as an intranet; orcombinations thereof, and may utilize a variety of networking protocolsnow available or later developed including, but not limited to,TCP/IP-based networking protocols.

FIG. 26 is a block diagram of electronic components 2600 of the end-userdevices 106 that are capable of executing instructions to implement themethod methods of FIGS. 16-22, and 24 . The end-user devices 106 mayinclude computers, smart phones, tablets, optical head-mounted displays,augmented reality smart glasses, and/or virtual reality headsets, etc.In the illustrated example, the electronic components 2600 include aprocessor 2602, memory 2604, storage 2606, input devices 2608, networkmodules 2610, a display 2612, a front-facing camera 2614, a back-facingcamera 2616, speakers 2618, and/or a GPS receiver 2620. The end-userdevices 106 may include other sensors (e.g., altimeter, magnetometer,accelerometer, LiDAR, RADAR, etc.) depending on the type of the end-userdevice 106.

The processor or controller 2602 may be any suitable processing deviceor set of processing devices such as, but not limited to: amicroprocessor, a controller-based platform with multiple processingcores, a suitable integrated circuit, one or more field programmablegate arrays (FPGAs), and/or one or more application-specific integratedcircuits (ASICs). In the illustrated example, the processor 2602 isstructured to include the example query generator 1400.

The memory 2604 may be volatile memory (e.g., RAM, which can includenon-volatile RAM, magnetic RAM, ferroelectric RAM, and any othersuitable forms); non-volatile memory (e.g., disk memory, FLASH memory,EPROMs, EEPROMs, memristor-based non-volatile solid-state memory, etc.),unalterable memory (e.g., EPROMs), and/or read-only memory. In someexamples, the memory 2604 includes multiple kinds of memory,particularly volatile memory and non-volatile memory. The storage 2606includes high-capacity storage devices, such as hard drives and/or solidstate drives, etc. The memory 2604 and/or the storage 2606 may storeinformation used by the ARPE 100 and the end-user device 106 to identifythe user and maintain and/or establish a session.

The memory 2604 and storage are computer readable media on which one ormore sets of instructions, such as the software for operating themethods of the present disclosure can be embedded. The instructions mayembody one or more of the methods or logic as described herein. In aparticular embodiment, the instructions may reside completely, or atleast partially, within any one or more of the memory 2604, the storage2606, and/or within the processor 2602 during execution of theinstructions.

The input devices 2608 are any suitable devices to facilitate receivinginput of queries 104, browse query results 102, and select of elementson the interfaces of FIGS. 6-11 . The input devices 2608 include, forexample, a microphone, the cameras 2614 and 2616, a keyboard, a button,a mouse, a touch screen, a controller, a gesture recognition device, agaze detection device, and/or a voice recognition system.

The network modules 2610 include communication devices that facilitatewired or wireless communication to the ARPE 100 via the externalnetwork. The network modules 2610 may include communication controllersfor mobile networks (e.g., Global System for Mobile Communications(GSM), Universal Mobile Telecommunications System (UMTS), Long TermEvolution (LTE), LTE Advanced, Code Division Multiple Access (CDMA),etc.), wireless networks (e.g., WiMAX (IEEE 802.16m); local areawireless network (including IEEE 802.11 a/b/g/n/ac or others) and/orWireless Gigabit (IEEE 802.11ad), etc.), 5th generation (5G) wirelesssystem, and/or wired networks (e.g., an Ethernet network, etc.).

The display 2612 is any suitable device to display the interfaces ofFIGS. 6-11 to the user. The display 2612 may include a heads-up display,a monitor or screen display (e.g., a liquid crystal display (“LCD”), anorganic light emitting diode (“OLED”) display, a flat panel display, asolid state display, etc.), a virtual reality display (e.g., a monocularhead mounted display or a binocular head mounted display), a waveguidedisplay, a liquid crystal on silicon (LCOS), etc.

The cameras 2614 and 2616 capture still images and video. In theillustrated example, the electronic components 2600 include thefront-facing camera 2614 and the back-facing camera 2616. Thefront-facing camera 2614 is positioned on the end-user device 106 on thesame side as the display 2612 to capture images and video of, forexample, the user. In some examples, the front-facing camera 2614 isused to track the location of the eyes of the user. The back-facingcamera 2616 is positioned on the end-user device 106 on the oppositeside as the display 2612. In some examples, the back-facing camera 2616is used to track gestures of the users.

The GPS receiver 2620 provides the coordinates of the end-user device106. In some examples, the GPS receiver 2620 includes an inertialnavigation system. The GPS receiver 2620 may also include anaccelerometer to determine the orientation of the end-user device 106.For example, the GPS receiver 2620 may provide the orientation of theend-user device relative to the direction the camera(s) 2414 and 2416are pointing.

FIG. 27 depicts a law enforcement map interface 2700 presented by theend-user device 106 of FIG. 1 to facilitate generating the search query104 and presenting and interacting with the query results 102 related tolaw enforcement and emergency services (e.g., federal, state, county andmunicipal law enforcement, fire departments, ambulance services, etc.).Additionally, in some examples, the a law enforcement map interface 2700facilitates generates event reports to be processed by the augmentedreality platform entity (ARPE) 100 and entered into the augmentedreality platform (ARP) database 112. In some such examples, a portion ofthe ARP database 112 used to populated the law enforcement map interface2700 is segregated from the portion of the ARP database 112 used topopulated other interfaces (e.g., the map interface 1000 of FIG. 10 ,etc.). In the illustrated example, the map query interface 520 displaysa map 2702. The map may be selected via one of the result interfaces(e.g., interfaces 600, 700, and 800 of FIGS. 6, 7, and 8 , above). Thelaw enforcement map interface 2700 includes interactive indicia 2704 ofevents relating to law enforcement and/emergency service activities. Inthe illustrated example, the law enforcement map interface 2700 alsoincludes the scrubber track 606, the event points 608, and/or theposition marker 614 to facilitate navigating though the events (e.g., asrepresented by the interactive indicia 2704) with differentchronological time frames. The scrubber track 606, the event points 608,and/or the position marker 614 facilitate sorting and browsing theinformation chronologically in varying degrees of granularity. Forexample, the information may be browsed hourly, daily, monthly, and/oryearly, etc. The interactive indicia 2704 are chronologically sorted.Using the scrubber track 606, the event points 608, and/or the positionmarker 614, a user browses the interactive indicia 2704 chronologically.In such a manner, the user may reconstruct movements and/or visualize anorder of events.

In some examples, the interactive indicia 2704 is displayed on the lawenforcement map interface 2700 cumulatively by timestamp or discretelyby timestamp. For example, when displayed cumulatively by timestamp,sliding the position marker 614 to a “TODAY” position may display allthe interactive indicia 2704 that occurred that day and sliding theposition marker 614 to a “THIS WEEK” position may display all theinteractive indicia 2704 that occurred that week. As another example,when displayed discretely by timestamp, sliding the position marker 614to a “now” position may only display the current interactive indicia2704 and sliding the position marker 614 to a “10:00-10:00 AM” positionmay only display the interactive indicia 2704 that occurred from 10:00AM to 10:10 AM.

The interactive indicia 2704 represent events of interest that becomesearch query 104 when interacted with. For example, the interactiveindicia 2704 may represent a person, place, or event relevant to lawenforcement and/or emergency services activity. When a user interactswith (e.g., via a touch screen of the end-user device 106, etc.), thelaw enforcement map interface 2700 submits the search query 104 relatedto the interactive indicia 2704 to the ARPE 100. The law enforcement mapinterface 2700 provides the query results 102 in hierarchies (e.g., thehierarchies 702, 704, and 706 of FIG. 7 above) to facilitate the userinteracting with the query results 102.

To process the search query 104, the ARPE 100 is communicatively coupledto content database providers 108 that are specialized in lawenforcement and emergency service data. For example, the contentdatabase providers 108 may include gun registration databases, crimereport databases, cell phone tracking databases, closed-circuittelevision (CCTV) video databases, license plate recognition (LPR)databases, vehicle registration databases, drivers license databases,shot identification databases (e.g., ShotSpotter® etc.), governmentalalert databases (e.g., managed by the National Weather Service, etc.),police, fire, military, or other types of governmental report databases,body camera image databases, law databases, evidence databases, and/orpersonnel databases, etc. Additionally, in some examples, the ARPE 100is communicatively coupled to live-updating data sources, such asreal-time tip databases, emergency call databases (e.g., 911 calls,etc.), emergency alert databases (e.g., fire alerts, Amber alerts,etc.), vehicle and/or phone tracking databases, and/or police and/oremergency personnel dispatch databases, etc. Additionally, in someexamples, the ARPE 100 is communicatively coupled to an emergencydispatch system to forward emergency alerts and/or eyewitness alertswhen the alerts are received from the end-user devices 106.

In some examples, the interactive indicia 2704 represent moving objects,such as of police cars, cars, bicycles, individuals (e.g., via cellphone signal, etc.). This facilitates tracking objects of interestand/or knowing potential support resources before a user responds to anemergency call. Additionally or alternatively, in some examples, theinteractive indicia 2704 represent crime reports. Additionally oralternatively, in some examples, the interactive indicia 2704 representreports of gunshots. For example, using the position marker 614, aninvestigator may browse the history of a person, place, or event inconnection with the interactive indicia 2704 representing a gunshot. Asanother example, by interacting with a particular interactive indicia2704 representing a gunshot, a search query 104 is generated with aperson, place and/or event connected with the gunshot as the queryterm(s). Additionally or alternatively, in some examples, theinteractive indicia 2704 represent emergency calls (such as to 911,etc.) and emergency incidents (e.g., a fire, a traffic accident, etc.).For example, using the position marker 614, a user may chronologicallybrowse persons, places, and/or events related to the emergency call oremergency incident related to the event represented by the interactiveindicia 2704. Additionally or alternatively, in some examples, theinteractive indicia 2704 represent license plate numbers of interestflagged in a database detected by a LPR reader. For example, using theposition marker 614, a user may chronologically browse persons, places,and/or events related to the license plate represented by theinteractive indicia 2704.

Additionally or alternatively, in some examples, the interactive indicia2704 represent when body cameras equipped to law enforcement officersare turned off. For example, a precinct captain may used the lawenforcement map interface 2700 to track the state of the body camerasequipped to police officers in the relevant precinct and navigate, usingthe position marker 614 to analyze, chronologically, places and eventsin connection with the state of the police officers' body cameras.Additionally or alternatively, in some examples, the interactive indicia2704 represent social media posts regarding persons or events associatedwith a particular location. Additionally or alternatively, in someexamples, the interactive indicia 2704 represent general alerts, such asroad closures and protest locations, etc.

In some examples, when the ARPE 100 is communicatively coupled to anavigation and/or traffic database, the query result 102 includesdirections to be displayed on the map 2702 to provide a safe and/or fastroute to a destination (e.g., a location represented by the interactiveindicia 2704).

In some examples, the law enforcement map interface 2700 facilitates auser generating,via the query generator 1400, report of accidents oremergencies. Additionally or alternatively, in some examples, via thelaw enforcement map interface 2700, the user provides crime tips (e.g.,anonymously) and/or eye witness images and/or video. In some suchexamples, the GPS coordinates of the end-user device 106 and the currenttime are included in the report in order for the ARPE 100 to generate acorresponding interactive indicia 2704. In some examples, via the lawenforcement map interface 2700, a user reports violations of municipalcode (e.g., parking violations, etc.) in order for the ARPE 100 togenerate a corresponding interactive indicia 2704. In some suchexamples, when the query generator 1400 generates the event to send tothe ARPE 100, the query generator 1400 collects data from the inputdevices 2608 of the end-user device 106 to include in with the eventreport. For example, when the end-user device 106 includes a LiDAR orplenoptic camera, geometry, direction of camera, and/or viewing angle,etc. of the camera may be included in the event report.

In some examples, the law enforcement map interface 2700 includes anemergency mode that automatically generates emergency reports thatinclude GPS coordinates and/or images and/or video from a camera of theend-user device 106. In such examples, the law enforcement map interface2700 provides a notification when the emergency report is acknowledgedby the recipient. Additionally or alternatively, in some examples, thelaw enforcement map interface 2700 provides an interface to live streamvideo and/or audio from the end-user device 106 that may be indexed sothat other users can view/listen to the stream via interacting with acorresponding interactive indicia 2704.

In some examples, via the law enforcement map interface 2700, specificinteractive indicia 2704 are creatable to assign specific lawenforcement personnel, vehicles, support personnel and/or equipment to alocation and/or event represented by another interactive indicia 2704.

In some examples, the ARPE 100 tracks the location of the end-userdevice 106 and provides an alert to the end-user device 106 when theend-user device 106 leaves a defined area (such as a precinct or apatrol route, etc.).

In some examples, via the law enforcement map interface 2700, using theinput devices 2608 of the end-user device 106, the user provides data(e.g., dates, locations, events, persons, etc.) regarding evidence to beuploaded to the ARPE 100 and indexed into the ARP database 112.

In some examples, the law enforcement map interface 2700 facilitatesusing the input devices 2608 of the end-user device 106 to scan andverify an officer's ID (such as a badge with a badge number) to verifythe identity of the officer.

In some examples, the law enforcement map interface 2700 is organizedinto layers. Each layer is associated with different types or groups oftypes of the interactive indicia 2704. For example, a first layer may beassociated with interactive indicia 2704 representative of gunshots anda second layer may be associated with interactive indicia 2704representative of 911 calls. In such examples, the law enforcement mapinterface 2700 includes a layer navigation interface 2706 to facilitatechoosing which layer, and thus which interactive indicia 2704 aredisplayed on the law enforcement map interface 2700. In some suchexamples, multiple layers are selectable to display multiple types ofinteractive indicia 2704 on the law enforcement map interface 2700.

FIG. 28 illustrates the augmented reality interface 1102 of FIG. 11providing location specific sporting event content superimposed on liveimages captured by the end-user device 106 of FIG. 1 . The end-userdevice 106 provides its location to the ARPE 100 (e.g., via thepre-query data 1408). The ARPE 100 determines whether one of the contentdatabase providers 108 associated with a stadium corresponding to thelocation of the end-user device 106 is providing location specificcontent 2302. The location specific content 2302 is content that isbeing generated in near real-time (e.g., there may be a delay betweenthe creation of the content for processing and/or indexing, etc.) forusers located at the venue. For example, the location specific contentmay be closed captioning of an announcer and/or athlete tracking dataused to track the position of specific athletes 2802 on thefield/court/pitch. In some examples, the ARPE 100 provides the locationspecific content 2302 via the pre-query response 1410. In some suchexamples, as long as enabled, the ARP 100 continuously pushes thelocation specific content 2302 to the end-user device 106 via thepre-query response 1410.

If enabled, the location specific content 2302 is displayed by theaugmented reality query interface 522 by superimposing the locationspecific content 2302 on the image being captured by the camera(s)(e.g., the cameras 2614 and 2616 of FIG. 26 above) of the end-userdevice 106. In some examples, the location specific content 2302 isdisplayed on a specific highlighted and/or outlined object 1104. In theillustrated example, the location specific content 2302 includes closedcaptioning of announcements and/or commentary regarding the sportingevent. Additionally, in the illustrated example, the augmented realityinterface 1102 provides tracking of specific athletes 2802 and, uponinteraction with one of the athletes 2802 via, for example, a touchscreen of the end-user device 106, displays the abbreviated query panel1106 for topics associated with that athlete 2802. To facilitatealigning the athlete tracking data with the location of the athlete onthe screen of the end-user device 106, the end-user device providecamera field of view and orientation data to the ARPE 100. When the userinteracts with the abbreviated query panels 1106, the augmented realityquery interface 522 generates and sends the search query 104 regardingthe corresponding topic to the ARPE 100.

In some example, the location specific content 2302 includes videos,images, and/or audio that is accessible to the user while the end-userdevice 106 is with a region defined by a geofence (e.g., with a radiusof the stadium, etc.). This content may be timed and/or curated toprovide relevant information to the user while the sporting event is notcurrently active (e.g., during pre-game and/or half-time, etc.). In someexamples, this content is pushed to the end-user device 106 when such afeature is enabled.

In some examples, the augmented reality interface 1102 superimposesother information, such as the score, on the images captures by thecameras. In some such examples, the superimposed information becomes asearch query 104 when the user interacts with the information. In somesuch examples, the query results 102 are presented in hierarchies (e.g.,the hierarchies 702, 704, and 706 of FIG. 7 above) to facilitate theuser interacting with the query results 102.

In some examples, the augmented reality interface 1102 providesdirections to and from a venue superimposed on the images captures bythe cameras. Additionally, in some examples, the augmented realityinterface 1102 provides directions to a user's seat within the stadiumand/or to various concession stands within the stadium.

In some examples, the augmented reality interface 1102 may superimposeuser generated content onto the images captured by the camera based on ageotag in the content. For example, when a user in the stadium posts apublic social media post that is tagged with coordinates within thestadium, the augmented reality interface 1102 displays the social mediapost proximate those coordinates.

In some examples, the augmented reality interface 1102 provides acatalogue 2804 of items or services available at the stadium. Using theseat assignment and/or GPS coordinates, the ARPE 100 associates ordersand/or requests made via the catalogue 2804 to the location of theend-user device 106. In such a manner, the items and/or services orderedthrough the catalogue 2804 may be delivered to the user. Additionally oralternatively, through the augmented reality interface 1102, the useralerts the venue to an emergency and includes GPS coordinates, seatassignment, and/or data captured by the input devices 2608 of theend-user device 106 (e.g., video, images, audio, etc.). In some suchexamples, the ARPE 100 transforms the alert into an interactive indicia2704 on a law enforcement map interface 2700 used by emergencyresponders at the venue.

In some examples, the ARPE 100 is communicatively coupled to a contentdatabase provider 108 that provides real estate information (e.g.,purchase history, county recorder's history, crime statistics, relevantnews pertaining to the location, shopping opportunities. schooldistrict, etc.). In some examples, the ARPE 100 is communicativelycoupled to a content database provider 108 that provides ancestryinformation that associates locations (e.g., immigration locations,birth locations, death locations, grave locations, marriage locations,etc.) with people and dates. Additionally, the ancestry informationassociates people together in family trees. In some examples, the ARPE100 is communicatively coupled to a content database provider 108 thatprovides traffic data to facilitate providing traffic information inaugmented reality, virtual reality and/or mixed reality. In someexamples, the ARPE 100 is communicatively coupled to a content databaseprovider 108 that provides weather data to facilitate providing weatherinformation in augmented reality, virtual reality and/or mixed reality.In some examples, the weather data includes geometry data that providesa three dimensional construct of the weather that can be incorporatedinto one of the interfaces discussed above.

In some examples, the ARPE 100 is communicatively coupled to a contentdatabase provider 108 that provides survey and/or election result data.In some examples, the ARPE 100 is communicatively coupled to a contentdatabase provider 108 that provides census data. In some examples, theARPE 100 is communicatively coupled to a content database provider 108that provides financial data (e.g., stock market data, etc.). In someexamples, the ARPE 100 is communicatively coupled to a content databaseprovider 108 that provides flight navigation data to track, for example,aircraft and/or unmanned aerial vehicles in the law enforcement mapinterface 2700. In some examples, the ARPE 100 is communicativelycoupled to a content database provider 108 that provides academic data(e.g., published papers in journals, etc.), scientific data, and/orresearch data. In some examples, the ARPE 100 is communicatively coupledto a content database provider 108 that provides agricultural data(e.g., soil data, weather data, crop data, fertilizer use data, plantinghistory data, etc.).

In some examples, the ARPE 100 is communicatively coupled to a contentdatabase provider 108 that provides medical data and/or hospital data(e.g., patient location and health data, patient geometry data, etc.) tofacilitate patient care in a medical facility. For example, a hospitalfloor plan may the map in the map interface 1000. In such an example,the interactive indicia 2704 may be generated for patients and clinicalevents. As another example, medical data, such as X-rays or CAT scanscan be superimposed onto images captured by the camera in the augmentedreality query interface 522 using the patient geometry data tofacilitate viewing the medical data on the patient.

In some examples, the map query interface 520 is used as a fleetmanagement tool where locations and timestamps for vehicles aretransformed into the interactive indicia 2704 to be displays andchronologically tracked. Additionally, the user may create specificevents to direct vehicles to locations at certain times so that thoseplanned events are displayed in the map query interface 520 of, forexamples, drivers and/or other fleet management personnel.

In this application, the use of the disjunctive is intended to includethe conjunctive. The use of definite or indefinite articles is notintended to indicate cardinality. In particular, a reference to “the”object or “a” and “an” object is intended to denote also one of apossible plurality of such objects. Further, the conjunction “or” may beused to convey features that are simultaneously present instead ofmutually exclusive alternatives. In other words, the conjunction “or”should be understood to include “and/or”. The terms “includes,”“including,” and “include” are inclusive and have the same scope as“comprises,” “comprising,” and “comprise” respectively.

The above-described embodiments, and particularly any “preferred”embodiments, are possible examples of implementations and merely setforth for a clear understanding of the principles of the invention. Manyvariations and modifications may be made to the above-describedembodiment(s) without substantially departing from the spirit andprinciples of the techniques described herein. All modifications areintended to be included herein within the scope of this disclosure andprotected by the following claims.

What is claimed is:
 1. A method comprising: presenting, on a display ofan end-user device, live image data being captured by a camera of theend-user device; determining a pose of the end-user device, the poseincluding a location of the end-user device; sending the pose of theend-user device to an augmented reality platform entity; receivingclosed captioning data from the augmented reality platform entity, theclosed captioning data based on the location of the end-user device andincluding an indication of a first topic embedded within the closedcaptioning data; and superimposing the closed captioning dataincorporating an interactive element associated with the first topiconto the live image data, wherein the first topic includes a query panelassociating the first topic with the interactive element.
 2. The methodof claim 1, including highlighting objects of interest in the displayedlive image data being captured by the camera, the objects of interestassociated with a topic.
 3. The method of claim 2, wherein highlightingthe objects of interest in the displayed live image data being capturedby the camera includes: sending the pose of the end-user device to anaugmented reality platform entity, the pose including the location andorientation of the end-user device; receiving the objects of interestfrom the augmented reality platform entity based on the pose of theend-user device.
 4. The method of claim 2, wherein superimposing theclosed captioning data onto the live image data includes superimposingthe closed captioning data onto one of the objects of interest.
 5. Themethod of claim 2, including in response to receiving a selection of oneof the objects of interest associated with second topics, sending aquery to the augmented reality platform entity that includes thecorresponding one of the second topics.
 6. The method of claim 5,including, in response to receiving a query result from the augmentedreality platform entity: presenting a scrubber track, a scrubber, and aplurality of event points, the plurality of event points indicativechronological categories associated with the query result; anddisplaying a first hierarchy of a plurality of hierarchies associatedwith records included in the query result associated with one of thechronological categories selected by a position of the scrubber on thescrubber track corresponding to an associated one of the plurality ofevent points.
 7. The method of claim 1, including in response toreceiving a selection of the interactive element within the displayedclosed captioning data, sending a query to the augmented realityplatform entity that includes the corresponding first topic.
 8. Themethod of claim 1, wherein receiving the closed captioning data from theaugmented reality platform entity includes receiving the closedcaptioning data in response to being within an area defined by ageofence.
 9. The method of claim 1, wherein content of the closedcaptioning data is based on the location of the end-user device and apose of the end-user device.
 10. An end-user device comprising: memorywith an application; and a processor communicatively coupled to thememory, the application, when executed, causing the processor to:present, on a display, live image data being captured by a camera of theend-user device; determine a pose of the end-user device, the poseincluding a location of the end-user device; send the pose of theend-user device to an augmented reality platform entity; receive closedcaptioning data from the augmented reality platform entity, the closedcaptioning data based on the location of the end-user device andincluding an indication of a first topic embedded within the closedcaptioning data; and superimpose the closed captioning dataincorporating an interactive element associated with the first topiconto the live image data, wherein the first topic includes a query panelassociating the first topic with the interactive element.
 11. Theend-user device of claim 10, wherein the application, when executing,causes the processor to highlight objects of interest in the displayedlive image data being captured by the camera, the objects of interestassociated with a topic.
 12. The end-user device of claim 11, wherein tohighlight the objects of interest in the displayed live image data beingcaptured by the camera, the application, when executing, causes theprocessor to: send the pose of the end-user device to an augmentedreality platform entity, the pose including the location and orientationof the end-user device; receive the objects of interest from theaugmented reality platform entity based on the pose of the end-userdevice.
 13. The end-user device of claim 11, wherein to superimpose theclosed captioning data onto the live image data, the application, whenexecuting, causes the processor to superimpose the closed captioningdata onto one of the objects of interest.
 14. The end-user device ofclaim 11, wherein the application, when executing, causes the processorto, in response to receiving a selection of one of the objects ofinterest associated with second topics, send a query to the augmentedreality platform entity that includes the corresponding one of thesecond topics.
 15. The end-user device of claim 10, wherein theapplication, when executing, causes the processor to, in response toreceiving a selection of interactive element within the displayed closedcaptioning data, send a query to the augmented reality platform entitythat includes the corresponding first topic.
 16. The end-user device ofclaim 15, wherein the application, when executing, causes the processorto, in response to receiving a query result from the augmented realityplatform entity: present a scrubber track, a scrubber, and a pluralityof event points, the plurality of event points indicative chronologicalcategories associated with the query result; and display a firsthierarchy of a plurality of hierarchies associated with records includedin the query result associated with one of the chronological categoriesselected by a position of the scrubber on the scrubber trackcorresponding to an associated one of the plurality of event points. 17.A tangible computer readable medium comprising instructions that, whenexecuted, cause an end-user device to: present, on a display of anend-user device, live image data being captured by a camera of theend-user device; determine a pose of the end-user device, the poseincluding a location of the end-user device; send the pose of theend-user device to an augmented reality platform entity; receive closedcaptioning data from the augmented reality platform entity, the closedcaptioning data based on the location of the end-user device andincluding a visual indication of a first topic embedded within theclosed captioning data; and superimpose the closed captioning dataincorporating an interactive element associated with the first topiconto the live image data, the visual indication linking the first topicwith the interactive element.
 18. The tangible computer readable mediumof claim 17, wherein the visual indication includes a query panel.